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Les  systemes  multi-senseurs  et  le  fusionnement  des 
donnees  pour  les  telecommunications, 
la  teledetection  et  les  radars 

(AGARD  CP-595) 


Synthese 

Le  fusionnement  des  donnees  est  une  science  qui  a  intrigue  l’homme  depuis  ses  origines.  Au  cours  des  siecles,  differentes  theories  et 
techniques  ont  ete  developpees,  mais  la  cadence  de  ce  developpement  n’a  jamais  atteint  celle  de  la  demiere  decennie,  que  1  on  ne 
peut  que  qualifier  d’explosive.  Ce  symposium  SPP,  tenu  a  Lisbonne,  au  Portugal,  a  permis  de  presenter  des  methodes  et  des 
techniques  novatrices  qui  temoignent  de  la  large  mise  en  application  de  la  science  du  fusionnement  des  donnees  dans  bon  nombre  de 
systemes  militaires. 

Les  communications  presentees  lors  du  symposium  ont  couvert  les  quatre  grands  domaines  techniques  suivants: 

—  les  multisenseurs  et  leurs  applications 

—  les  methodes  de  fusionnement  des  donnees 

—  les  systemes  de  detection  et  les  techniques  de  fusionnement 

—  la  validation  operationnelle  et  experimentale  et  les  technologies  associees 

Le  communications  presentees  ont  temoigne  des  progres  spectaculaires  realises  dans  la  comprehension  des  sciences,  techniques  et 
technologies  qui  sous  tendent  le  fusionnement  des  donnees.  L’importance  des  bases  de  donnees  et  des  banques  de  donnees  a  ete 
soulignee  par  plusieurs  conferenciers.  Un  certain  nombre  de  parametres  strategiques,  determinants  pour  le  fusionnement  des 
donnees,  ont  ete  consideres  comme  etant  d’une  importance  particuliere,  a  savoir: 

—  P opportunity  du  choix  de  senseurs,  leur  nombre  et  F obligation  d’independance  liee  aux  calculs  qu’ils  effectuent; 

—  la  fafon  de  constituer  la  base  de  donnees  et 

—  l’importance  de  bien  comprendre  les  limitations  des  observations  effectuees  par  les  differentes  categories  de  senseurs 

Suite  aux  discussions  qui  ont  eu  lieu,  il  est  apparu  tres  clairement  que  ces  questions  ne  sont  pas  encore  resolues  et  qu  il  reste 
beaucoup  a  faire  avant  de  pouvoir  definir  les  domaines  d’application.  A  titre  d’exemple,  ni  les  theories  classiques  comme  la  limite 
Cramer-Rao  pour  le  traitement  des  signaux,  ni  la  limite  de  Shannon  dans  la  theorie  de  l’information  ne  paraissent  adequates  pour  la 
definition  a  priori  de  Foptimisation  d’une  base  de  donnees. 

Une  fois  que  la  base  de  donnees  est  constitute,  le  fusionnement  de  donnees  peut  commencer.  Le  symposium  a  conclu  qu’en  general, 
chacune  des  methodes  presentees  donnait  des  resultats  satisfaisants  mais  non  pas  optimaux.  Des  methodes  classiques, 
convenablement  adaptees  et  perfectionnees,  ont  ete  appliquees  avec  succes.  Ces  methodes  comprennent  la  theorie  des  probabilites  et 
l’approche  Bayesienne,  en  plus  de  la  theorie  des  ensembles  flous  et  l’analyse  des  series  temporelles.  Certaines  methodes  plus 
recentes  presentent  de  l’interet  aussi.  Par  exemple,  ajouter  une  deuxieme  mesure  a  la  theorie  des  probabilites  offre  de  nouvelles 
possibilites.  De  la  meme  fa9on,  la  theorie  des  ondelettes  permet  de  faire  des  mesures  locales  en  generalisant  le  filtrage 
morphologique.  Ces  deux  theories  ont  ete  presentees  et  les  differents  degres  de  reussite  obtenus  ont  ete  demontres. 

Il  se  pourrait  que  d’autres,  telles  que  la  methode  globale  qui  est  citee  dans  la  communication,  ou  l’analyse  multifractale,  qui  pourrait 
etre  consideree  comme  la  continuation  de  la  theorie  des  ondelettes,  ou  bien  une  toute  nouvelle  theorie  encore  inconnue,  viennent 
enrichir  la  panoplie  d’approches  possibles.  Ces  methodes  sont  a  decouvrir,  implicitement,  parmi  les  outils  utilises  pour  le 
fusionnement  des  donnees,  comme  par  exemple  dans  le  cas  des  systemes  neuronaux. 

Tout  au  long  du  symposium,  les  conferenciers  ont  decrit  des  methodes  novatrices  qui  ont  conduit  a  des  applications  reussies, 
debouchant  sur  des  systemes  qui  ont  atteint  ou  qui  sont  en  passe  d’atteindre  leur  capacite  operationnelle  optimale.  Ces  applications 
concement  quatre  domaines  tres  importants,  a  savoir: 

—  l’imagerie 

—  les  systemes  radar,  F  identification  et  la  poursuite 

—  les  telecommunications 

—  les  aides  a  la  prise  de  decisions 

Comme  il  est  souvent  le  cas  dans  un  symposium  consacre  a  des  questions  relevant  des  demiers  developpements  d  une  science 
dynamique,  bon  nombre  des  participants  auront  ete  amenes  a  considerer  ce  sujet  d’une  fafon  nouvelle,  a  la  lumiere  des 
commentaires  approfondis  des  auteurs.  Dans  la  mesure  ou  il  est  possible  de  juger  deja  de  la  reussite  du  symposium,  nous  sommes 
convaincus  qu’il  sera  considere  comme  tres  fructueux. 

Les  resultats  presentes  a  Lisbonne  montrent  tres  clairement  que  cette  science  prometteuse  s’ affine,  et  la  diversite  de  ses  applications 
indique  qu’elle  est  d’un  interet  universel.  En  particulier,  en  ce  qui  conceme  les  applications  militaires,  la  surveillance,  la  detection  et 
Foptimisation  des  aides  a  la  decision  sont  des  domaines  qui  suscitent  de  plus  en  plus  d  interet.  L  influence  de  cette  science  sur  la 
formulation  des  strategies  futures  au  sein  du  secteur  militaire  et  en  dehors  est  desormais  incontestable. 

Nous  esperons  que  ce  symposium  SPP  conduira  a  l’organisation  d’autres  conferences  afin  de  permettre  de  poursuivre  1  exploration 
de  ce  sujet  riche  en  technologie.  Pour  le  moment,  nous  nous  bomerons  a  exprimer  notre  satisfaction  devant  la  qualite  des 
communications  et  des  discussions  qui  ont  animees  ce  symposium,  et  a  souligner  sa  pertinence  pour  les  pays  membres  de  l’OTAN. 
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Multi-Sensor  Systems  and  Data  Fusion  for 
Telecommunications,  Remote  Sensing  and  Radar 

(AGARD  CP-595) 

Executive  Summary 

Data  fusion  is  a  science  that  has  intrigued  mankind  since  his  origins.  Over  time,  theories  and  techniques  have  continued  to  develop 
but  never  at  a  pace  which,  in  the  past  decade,  can  be  described  only  as  explosive.  This  SPP  symposium,  held  in  Lisbon,  Portugal,  has 
revealed  innovations  in  methods  and  technologies  leading  to  broad  implementation  of  the  science  in  a  number  of  systems, 
particularly  for  this  symposium,  in  the  military  area. 

This  symposium  consisted  of  papers  which  were  classified  into  four  broad  technical  areas: 

—  Multisensors  and  their  applications. 

—  Data  fusion  methods 

—  Sensors  systems  and  fusion  techniques. 

—  Operational  and  experimental  validations  with  their  associated  technologies. 

The  papers  presented  demonstrated  the  dramatic  progress  recently  in  the  our  understanding  of  the  underlying  science,  technologies, 
and  techniques  of  data  fusion.  The  importance  of  data  bases  and  data  banks  were  underscored  by  many  speakers.  Of  particular 
importance  were  the  several  strategic  parameters  critical  to  data  fusion: 

The  appropriateness  of  the  choice  of  the  sensors,  their  number  and  the  necessity  for  independence  that  their  measurements 
represent; 

—  The  manner  in  which  the  data  bank  is  constituted;  and 

—  The  importance  of  an  understanding  of  the  limitations  of  observations  from  classes  of  sensors. 

It  was  clear  from  the  discussions  that  these  topics  are  not  yet  resolved  and  that  much  remains  to  be  done  before  the  areas  for 
application  can  be  fully  realized.  For  example,  neither  the  classical  theories  as  a  bound  of  Cramer-Rao  in  processing  of  the  signals  or 
the  bound  of  Shannon  in  the  theory  of  the  information  appear  sufficient  to  define,  a  priori,  the  optimization  of  a  database. 

Once  the  data  base  constituted,  fusion  of  the  data  can  be  undertaken.  A  general  theme  of  the  symposium  was  that,  in  general,  the 
methods  presented  each  gave  satisfactory,  though  not  optimum,  results.  Classical  methods,  adapted  and  perfected,  have  been  applied 
with  a  much  success.  These  included  the  theory  of  probabilities  and  the  Bayesian  approach  in  addition  to  the  theory  of  the  fuzzy  sets 
and  the  analysis  of  temporal  series.  More  recent  methods  also  seem  attractive.  For  example,  the  theory  of  probabilities,  by  the 
introduction  of  a  second  measure,  offers  new  possibilities,  and  wavelets  theory  that  allows  local  measures  by  generalizing  the 
morphological  filtering,  were  presented  and  in  each  case,  demonstrated  varying  degrees  of  success. 

One  can  wonder  if  other  methods  such  as  the  method  of  aggregates  that  has  been  mentioned  in  communication,  or  the  multifractal 
analysis  that  could  be  seen  as  a  continuation  of  the  wavelets  theory,  or  even  some  yet  undisclosed  new  theories,  might  not,  in  the 
near  future,  enlarge  the  panoply  of  approaches.  Implicitly,  one  discovers  these  methods  among  the  tools  used  in  data  fusion,  as  for 
example,  in  the  case  of  neural  systems. 

Throughout  the  symposium,  the  innovative  methods  described  by  the  authors  have  led  to  applications  that  allowed  for  successful 
demonstrations  and  systems  that  are  nearing  or  are  at  full  operational  capability.  These  applications  focused  on  four  very  important 
areas: 

—  imagery. 

—  radar  systems,  identification,  and  tracking. 

—  telecommunications. 

—  assistance  to  decision  making. 

As  is  often  the  case  of  a  symposium  dedicated  to  a  topic  on  the  cutting  edge  of  a  dynamic  science,  many  of  the  attendees  will  have 
been  challenged  to  think  in  new  ways  by  the  thoughtful  comments  of  the  authors.  To  the  extent  that  early  estimates  of  this  can  be 
made,  we  are  confident  this  symposium  will  be  proven  to  have  been  particularly  fruitful. 

The  results  presented  in  Lisbon  clearly  show  the  extent  this  promising  science  is  maturing  and  the  diversity  of  applications  shows 
that  it  s  relevance  is  universal.  Specifically  in  the  area  of  military  applications,  surveillance,  detection,  and  the  optimization  of 
decision-making  are  domains  that  are  receiving  increasing  attention.  The  influence  of  this  science  on  future  strategies  in  and  out  of 
the  military  arena  is  no  longer  in  doubt. 

It  is  our  hope  that  this  SPP  symposium  will  lead  to  other  conferences  in  which  the  exploration  of  this  technologically-rich  subject 
will  continue.  For  the  present  symposium,  it  is  sufficient  to  admit  satisfaction  in  the  quality  of  presentations  and  discussions  and  the 
relevance  that  it  represents  for  the  NATO. 


iii 


Contents 


Page 

Executive  Summary  ill 

Synthese  iv 

Theme/Theme  ix 

Sensor  and  Propagation  Panel  x 

Reference 

TA10  -  Advanced  Information  Processing  for  Multisensory  Systems  1 

I.E.P.G.  Cooperative  Technological  Programs  Technological  Area  10 

by  C.  de  Leeuw  and  V.  Chalmeton 

Aircraft  Sensor  Data  Fusion:  An  Improved  Process  and  the  Impact  of  ESM  Enhancements  2 

by  C.A.  Noonan  and  M.  Pywell 

Multi-Sensor  Remote  Sensing  for  Military  Cartography  3 

by  A.C.  van  den  Broek,  P.  Hoogeboom  and  M.  van  Persie 

Design  of  a  Multi-Sensor  System  for  3D  Site  Model  Acquisition  and  Exploitation  4 

by  I.  Tannous,  S.  Gobert,  T.  Laurengot,  J.-M.  Dulac  and  O.  Goretta 

Airborne  Dual  Sensor  mmW-Signatures  of  Maritime  Targets  and  Sea-Clutter  5 

by  R.  Makaruschka  and  H.  Essen 

Data  Fusion  for  Long  Range  Target  Acquisition  6 

by  P.  Verlinde,  D.  Borghys,  C.  Pemeel  and  M.  Acheroy 

Paper  7  Withdrawn 

Two-Frequency  Millimeter  Wave  SAR  8 

by  H.  Schimpf  and  H.  Essen 

Systeme  Multi-Senseurs  Radar  et  Optronique  pour  l’Attaque  au  Sol  9 

by  E.  Mignot,  J.P.  Mestre  and  D.  Levaillant 

IR-Signatures  of  Ships:  A  Classification  Feature  Within  a  Sensor  Network  10 

by  H.D.  vom  Stein,  J.  Gunther  and  K.  Haese 

Advanced  Electro  Optic  Targeting  System  11 

by  R.W.  Jacobs  and  H.S.  Lapp 

Paper  12  Withdrawn 

Methode  devaluation  des  Performances  d’une  Poursuite  Multicapteur  13 

by  M.  Desbois 


V 


Reconnaissance  de  scenes  dynamiques  a  partir  de  donnees  issues  de  capteurs:  le  projet  14 

PERCEPTION 

by  C.  Tessier 

Multisensor  Data  Fusion  for  Automatic  Recognition  of  High  Value  Surface  Targets  15 

by  K.-H.  Bers,  H.  Essen,  K.  Jager  and  H.  Schimpf 

A  KF-Based  Integration  System  for  Land  Vehicle  Tracking  from  Real  DGPS  and  INS  Data  16 

by  C.S.  Regazzoni,  A.  Teschioni  and  G.  Tacconi 

ARTAS:  Multisensor  Tracking  in  an  ATC  Environment  17 

by  R.A.  Hogendoom  and  W.H.L.  Neven 

Low  Altitude  Wave  Propagation  Effects  in  a  Transmitter  Independent  Receiver  Network  18 

by  N.J.  Farsaris  and  P.P.  Stavroulakis 

Non-Linear  Prediction  of  Ionospheric  Parameters  Important  for  Forecasting  the  19 

Battlespace  Environment 

by  N.M.  Francis,  A.G.  Brown,  A.  Akram,  P.S.  Cannon  and  D.S.  Broomhead 

Fusion  de  donnees  et  fiabilite  des  transmissions  HF  20 

by  C.  Goutelard,  J.  Caratori  and  C.  Pautot 

Fusion  de  donnees  et  sondeur  a  extremement  faible  puissance  pour  telecommunications  21 

by  C.  Goutelard  and  C.  Pautot 

HF  Transmitter  Location  using  Super-resolution  DF  and  Ionospheric  Sounding  22 

by  H.J.  Strangeways 

Smart  Antenna  Array  Processing  for  Spread  Spectrum  Wireless  Communication  Systems  23 

by  Y.F.  Huang.  S.  Kapoor,  S.  Gollamudi  and  S.  Nagaraj 

Propagation  Diversity  Constraints  on  Reception  of  GPS  Signals  at  High  and  Equatorial  24 

Latitudes 

by  J.  Aarons  and  B.  Lin 

Meteorological  Data  Fusion  to  Assess  EM/EO  Propagation  Conditions  in  Coastal  25 

Environments 

by  J.  Rosenthal,  R.  Helvey,  P.  Greiman  and  M.  McGovern 

Utilisation  d’ Informations  Contextuelles  dans  des  Algorithmes  de  Pistage  Multicapteur  26 

by  V.  Nimier 

SAGESSE:  un  modele  de  representation  de  donnees  pour  la  Fusion  de  Donnees  27 

Symboliques 

by  Y.  Pollet  and  S.  Robidou 

Adaptive  Intensity  Matching  Filters:  A  New  Tool  for  Multi-Resolution  Data  Fusion  28 

by  S.  de  Bethune,  F.  Muller  and  M.  Binard 

The  Fusion  of  Organic  and  Non-Organic  Identity  Information  Sources  Using  Evidential  29 

Reasoning 

by  E.  Bosse,  J.  Roy  and  S.  Paradis 

Pixel  Fusion  for  Roads  Detection  and  Classification  30 

by  S.  Fabre,  X.  Briottet,  P.  Marthon  and  A.  Appriou 


vi 


Sensor  Synergetics:  The  Design  Philosophy  of  the  Sensor  Fusion  Demonstrator  and  31 

Testbed  (SFD)  at  TNO-FEL 

by  AJ.  van  der  Wal 

The  Nemesis  Identification  Data  Fusion  Demonstrator  32 

by  P.N.  Griffith,  L.A.  Hooper  and  W.M.  Everitt 

Communications  Management  in  Battlespace  Data  Fusion  33 

by  R.H.  Deaves,  D.  Nicholson,  P.  Greenway  and  P.  Vangasse 

NATO  Alliance  Ground  Surveillance  Interoperability  34 

by  PJ.  Lenk  and  G.  Retzer 

Imagery  and  Signals  Intelligence  Data  Fusion:  Issues  and  Methods  35 

by  E.L.  Waltz  and  J.  Hart 

NOSTRAMARINE:  Un  Concept  de  Detection  Multistatique  Adapte  a  la  Surveillance  des  36 

Cibles  Basse  Altitude 

by  M.  Lesturgie  and  M.  Flecheux 

The  NATO  Data  Fusion  Demonstrator  Project  37 

by  J.M.  Skinner 

Pixelless  Infrared  Imaging  Using  Quantum  Wells  38 

by  GJ.  Brown  and  F.  Szmulowicz 

Nonlinear  Optical  Frequency  Conversion  Materials  for  Multi-Band  Remote  Sensing  and  39 

Surveillance 

by  S.  Caracci,  N.  Femelius,  M.  Ohmer  and  D.  Zelmon 

Fusion  d’informations  et  reseaux  physiques  pour  les  systemes  multi-senseurs  sol-air  40 

by  C.  Nahum 

Development  of  a  Wideband  Airborne  Laser  Data  Link  41 

by  R.A.  Gill  and  R.J.  Feldmann 

Fusion  of  Visible  and  Thermal  Imagery  Improves  Situational  Awareness  42 

by  A.  Toet,  J.K.  IJspeert,  A.M.  Waxman  and  M.  Aguilar 


vii 


Theme 

Future  weapon  systems,  whether  ground,  air  or  space  based,  will  have  to  operate  efficiently  in  a  highly  complex,  hostile  and 
dynamic  battlespace  environment.  In  this  environment,  a  broad  array  of  deception  techniques  (ECM)  may  be  employed,  and 
the  need  to  fully  discriminate  among  a  dense  array  of  objects  will  be  increasingly  difficult.  In  the  past,  emphasis  has  been 
placed  on  optimizing  individual  sensors  for  specific  battlespace  functions.  Present  and  future  threats  will  demand  more 
capabilities  than  can  be  delivered  by  these  individual  sensors. 

However,  it  is  likely  that  when  data  from  many  individual  sensors  are  combined  in  an  intelligent  and  perhaps  adaptive 
manner  such  a  combination  may  provide  the  commander  information  needed  to  counter  these  threats.  By  using  data  from 
multiple  sensors,  operating  in  independent  domains  which  are  often  sensitive  to  widely  different  physical  phenomena,  the 
threats  posed  by  deception  techniques  may  be  better  neutralized.  Benefits  to  military  systems  can  be  anticipated  in  the  areas 
of  communication,  target  acquisition,  target  identification  and  delivery  accuracy,  together  with  overall  system  integrity  and 
robustness. 

In  recent  years,  significant  advances  have  been  made  in  data  fusion  techniques  and  algorithms.  Combined  with  the 
exponential  growth  of  available  computing  power  and  memory  density,  real-time  multisensor  data  fusion  is  now  becoming 
feasible  and  affordable.  Integrated  C3I  will  permit  an  architecture  applicable  to  efficient  data  exchange  and  may  exploit  the 
dramatic  increase  in  computing  power  now  available. 

Multisensor  data  fusion  can  be  applied  to  a  variety  of  systems  appropriate  to  communications,  surveillance  and  target 
recognition.  Data  from  multiple  sensors  will  lead  to  an  improved  and  more  complete  picture  of  the  battlespace  environment  - 
e.g.,  location,  heading,  strength,  etc.  of  both  friendly  and  hostile  forces.  Early  applications  of  data  fusion  techniques  in 
communications  have  been  employed  for  some  time.  For  example,  in  cellular  mobile  radio  or  monofrequency  broadcasting 
systems,  diversity  is  used  to  improve  reliability.  At  a  more  advanced  level,  the  reliability  of  data  transmissions  requires  a 
characterization  of  transmission  channels  which  results  from  measurements  (ground  and  space  based)  of  electron  density, 
magnetic  fields,  meteorological  conditions,  solar  activity,  etc.  Additional  information  can  be  extracted  from  waveforms  using 
data  fusion.  This  may  provide  a  robust  system  while  increasing  data  flow. 


Theme 

Les  systemes  d’armes  futurs,  qu’ils  soient  aeroportes,  spatioportes  ou  bases  au  sol,  devront  etre  exploites  dans  un  theatre  de 
bataille  dont  l’environnement  sera  hostile,  evolutif  et  extremement  complexe.  Une  large  gamme  de  techniques  de  deception 
(ECM)  pourra  etre  mise  en  ceuvre  et  la  discrimination  de  la  cible  dans  une  concentration  dense  d’objets  deviendra  de  plus  en 
plus  difficile.  Dans  le  passe,  Taccent  etait  mis  sur  l’optimisation  de  senseurs  individuels  destines  a  remplir  certaines  fonctions 
specifiques  du  theatre  d’ operations.  Or,  contrer  la  menace  actuelle  et  future  exige  plus  de  capacites  que  les  senseurs 
individuels  ne  peuvent  foumir. 

Cependant,  il  est  vraisemblable  que  lorsque  les  donnees  foumies  par  un  certain  nombre  de  senseurs  individuels  seront 
combinees  de  fa§on  intelligente  et,  peut  etre  adaptative,  la  combinaison  qui  en  resultera  foumira  aux  commandeurs  militaires 
les  informations  demandees  pour  contrer  ces  menaces.  II  se  pourrait  qu’en  exploitant  les  donnees  foumies  par  de  multiples 
senseurs,  fonctionnant  dans  les  domaines  independants,  sensibles  a  des  phenomenes  physiques  tres  diverses,  les  menaces 
posees  par  les  differentes  techniques  de  deception  soient  plus  faciles  a  neutraliser.  Des  avantages  pour  les  systemes  militaires 
sont  envisageables  dans  les  domaines  suivants:  les  communications,  l’acquisition  et  1’ identification  de  la  cible,  la  precision  du 
tir,  la  robustesse  et  l’integrite  globale  du  systeme. 

Ces  demieres  annees,  des  avancees  remarquables  ont  ete  realisees  dans  le  domaine  des  algorithmes  et  des  techniques  de 
fusionnement  des  donnees.  Avec  la  croissance  exponentielle  de  la  puissance  de  calcul  et  de  la  densite  memoire,  le 
fusionnement  des  donnees  multisenseur  en  temps  reel  devient  faisable  et  abordable.  Le  C3I  permettra  la  realisation 
d’architectures  qui  autoriseront  un  echange  de  donnees  performant  et  il  foumira  sans  doute  le  moyen  d’exploiter  la  croissance 
spectaculaire  de  la  puissance  de  calcul  qui  est  actuellement  disponible. 

Le  fusionnement  des  donnees  multisenseur  s’applique  a  un  eventail  de  systemes  qui  relevent  des  communications,  de  la 
surveillance  et  de  la  reconnaissance  de  la  cible.  En  ce  qui  conceme  la  surveillance  et  la  reconnaissance  de  la  cible,  Tapport  de 
donnees  multisenseur  se  traduira  par  une  representation  plus  complete  de  l’environnement  du  champ  de  bataille,  c’est-a-dire 
position,  direction  el  importance  des  forces  amies  et  ennemies.  Les  premieres  applications  des  techniques  de  fusionnement 
des  donnees  sont  en  service  depuis  un  certain  temps.  Par  exemple,  dans  les  systemes  de  radiodiffusion  monofrequence  ou 
dans  les  radios  mobiles  cellulaires  la  technique  de  diversite  est  utilisee  pour  accroitre  la  fiabilite.  A  un  niveau  plus  avance,  la 
fiabilite  de  la  transmission  des  donnees  passera  par  la  caracterisation  des  voies  de  transmission,  qui  sera  a  son  tour  tributaire 
de  la  mesure  (terrestre  et  spatiale),  de  la  densite  des  electrons,  des  champs  magnetiques,  des  conditions  meteorologiques,  de 
l’activite  solaire  etc.  Des  informations  complementaires  pourront  etre  extraites  des  formes  d’onde  par  le  fusionnement  des 
donnees.  Cette  approche  permettra  de  realiser  un  systeme  robuste,  tout  en  augmentant  le  flux  de  donnees. 
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SUMMARY 

Real-time  interpretation  of  large  amounts  of  battlefield  infor¬ 
mation  by  both  operator  and  system  is  increasingly  difficult 
due  to  steadily  growing  complexity,  and  integration  of  systems. 
Therefore  data  fusion,  i.e.  combining  data  from  several  sources 
in  order  to  obtain  a  global  and  coherent  view  on  the  battlefield, 
both  at  sensor  level  (multi  sensor  data  fusion)  and  at  abstract, 
strategic  information  level  (information  fusion),  becomes  more 
important. 

The  main  elements  of  the  TA10  demonstrator  project  are  (1) 
the  fusion  of  data  from  different  platforms  and  different  sen¬ 
sors,  (2)  the  management  and  the  allocation  of  the  platforms, 
drones,  helicopters,  and  sensors,  (3)  situation  assessment  and 
(4)  the  determination  of  the  requirements  for  real-time  data 
fusion.  To  perform  the  fusion  task,  the  required  input  consists 
of  plots/tracks  from  individual  sensors  (including  false  alarms) 
and  information  from  other  sources,  e.g.  intelligence,  and  geo¬ 
graphical  information.  TA10  focuses  on  the  integration  of 
information  from  different  platforms  (various  non-colocated 
platforms  and  drones)  and  different  sensors  (radar,  optronic 
and  ESM),  instead  of  using  only  the  sensors  on  a  single  plat¬ 
form.  TAlO’s  sensor  management  process  uses  the  results  from 
the  fusion  processes  in  order  to  allocate  resources  to  improve 
the  quantity  and  the  quality  of  the  information.  The  result  of 
the  project  is  a  demonstration  of  the  fusion  process,  the  sensors 
management  and  the  real-time  capabilities  using  simulated 
sensors.  The  TA10  demonstrator  project  therefore  covers  top¬ 
ics  like:  Reasoning  about  hypotheses,  alternatives,  uncertainty, 
lime  and  space;  Statistical  analyses;  Implementation  tech¬ 
niques,  such  as  functional  programming  and  knowledge-based 
technology. 

In  this  paper  we  will  describe  the  different  concepts  used  for 
the  multi  sensor  data  fusion  processes,  and  the  distributed 
asynchronous  software  architecture  chosen.  Furthermore,  the 
interaction  between  Data  Fusion,  Situation  Assessment  and 
Sensor/Platform  Management  will  be  explained  in  order  to 
illustrate  the  applied  technologies  for  advanced  information 
processing  for  multisensory  systems.  In  the  paper  some  of  the 
first  results  from  this  on-going  project  will  be  presented. 

THE  PARTICIPANTS 

The  project  is  set  up  as  an  international  cooperation  between 
industries  and  research  organisations  from  France  and  The 
Netherlands.  The  contributing  partners  are: 


-  Thomson-CSF/TCAR  (Velizy,  France) 

-  Matra  Systemes  &  Information  (Val  de  Reuil,  France) 

-  Hollandse  Signaalapparaten  (Hengelo,  The  Netherlands) 

-  TNO-FEL  (The  Hague,  The  Netherlands) 

-  NLR  (Amsterdam,  The  Netherlands) 


1.  SYSTEM  OVERVIEW 

The  primary  goal  of  this  project  is  to  design  and  implement  a 
demonstrator  showing  the  capabilities  of  multi  sensor  data 
fusion  through  advanced  information  processing.  The  TA10 
demonstrator  (TAD)  is  a  system  that  demonstrates  the  core  ele¬ 
ment  of  a  battlefield  ground  surveillance  system  by  simulation 
in  near  real-time.  The  core  element  of  the  battlefield  ground 
surveillance  system  consists  of  the  functional  parts:  Data 
Fusion,  Situation  Assessment,  Sensor/Platform  Management 
and  Sensor/Platform  simulation.  The  surveillance  system 
observes  the  real  world  through  a  non-collocated  heterogene¬ 
ous  multisensory  system.  The  perceived  world  is  organized 
into  a  world  model  by  using  appropriate  data  fusion  techniques 
in  order  to  represent  it  in  a  synthetic  way  to  the  operator.  The 
real  world  and  the  multiple  sensors  are  simulated  in  a  non¬ 
interactive  way.  So,  the  TAD  works  on  simulated  data  instead 
of  real  sensor  signals.  It  is  the  aim  of  the  study  to  define  and 
implement  a  demonstrator  able  to  show  the  use  of  advanced 
data  processing  techniques  on  data  fusion  under  (near)  real¬ 
time  constraints. 


Figure  1  -  Functional  partitioning  of  TAD  system 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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Figurel  identifies  the  main  functionality  of  the  TAD.  The 
architecture  itself,  based  upon  generic  principles,  is  described 
in  the  next  sections  and  will  show  that  the  functions  compris¬ 
ing  the  TAD  core  are  distributed  across  a  number  of  data  fusion 
and  sensor  management  nodes  due  to  the  geographical  distri¬ 
bution  of  the  sensor  platforms: 

1 .  The  Input  Simulator  generates  sensor  reports,  the  perceived 
truth,  as  result  of  battlefield  and  sensor/platform  simulation. 

2.  The  sensor  reports  are  processed  by  the  Data  Fusion  and 
Situation  Assessment  process  in  order  to  obtain  an  inferred 
representation,  i.e.  a  world  model. 

3.  Based  on  the  inferred  representation,  the  Data  Fusion  and 
Situation  Assessment  process  issues  Sensor/Platform 
requests  to  the  Sensor/Platform  Management. 

4.  These  requests  result  in  the  update  of  the  sensor  manage¬ 
ment  plan,  considering  sensor  performance  characteristics. 

5.  The  sensor  management  plan  will  be  the  basis  for  genera¬ 
tion  of  sensor/platform  commands  to  the  Input  Simulator, 
thereby  closing  the  control  and  information  loop. 

1.1  GENERIC  ARCHITECTURE 

The  TAD  relies  on  an  architecture  which  is  a  derivation  from  a 
generic  distributed  multi  sensor  data  fusion  architecture  (figure 
2  [6]).  From  the  resulting  definition  of  the  generic  architecture, 
the  demonstrator  architecture  (figure  3)  is  derived.  The  follow¬ 
ing  properties  emerge  from  this  approach: 

1.  The  generic  architecture  can  be  tailored  to  other  applica¬ 
tions  and  domains.  The  demonstrator  architecture  (figure  3 
and  4)  represents  such  a  specific  application  for  battlefield 
surveillance. 

2.  The  generic  architecture  is  reconfigurable,  which  means 
that  the  architectural  elements  (nodes)  and  interfaces  are 
defined  in  such  way  that  new  architectures  (corresponding 
to  specific  fusion/processing  schemes)  can  be  easily  gener¬ 
ated  and  their  performance  evaluated. 

3.  The  generic  architecture  is  flexible  in  adopting  new  or  other 
sensors  and  new  or  other  platforms.  The  demonstrator 
architecture  relies  on  a  specific  set  of  platforms  and  sensors. 

These  considerations  imply  that  the  demonstrator  architecture 
is  a  specific  instance  of  the  generic  architecture.  Throughout 
the  design  and  development  of  the  demonstrator,  this  general¬ 
ity  and  flexibility  is  pursued.  The  demonstrator  design  pursues 
generic  interfaces,  processes  and  algorithms  as  much  as  possi¬ 
ble  and  is  only  specific  there  where  it  is  unavoidable.  In  partic¬ 
ular,  the  fusion  nodes  have  a  common  structure,  which  is 
tailored  to  the  specific  function  they  perform. 


-  Abstract  output  of  f,  might  serve  as  input  for  fy 

-  fj,  SMjt  and  MMIj  might  be  co-Iocated  on  one  platform  pj 

Figure  2  -  Generic  distributed  multi-sensor  data  fusion 
architecture 


The  generic  DFSA  system  architecture  consists  of  the  follow¬ 
ing  elements: 

-  A  number  of  platforms  pj...  pk.  Each  platform  has  mounted 
a  number  of  sensors  s(p,l)...  s(p,m). 

-  A  number  of  data  fusion  nodes  fj...  fn  that  fuse  data  (from 
sensors,  i.e.  sensor/platform  reports,  and/or  from  other 
fusion  nodes,  i.e.  fusion  reports). 

-  A  situation  assessment  node,  that  takes  as  input  fusion 
reports  from  one  fusion  node  and  performs  situation  assess¬ 
ment.  It  also  makes  directives  for  the  management  of  the 
fusion  process  and  the  sensors/platforms. 

-  One  or  more,  probably  hierarchically  structured,  sensor/ 
platform  management  nodes  (hierarchical  control:  from 
general,  high  level  sensor  or  platform  directives  to  specific, 
low  level  sensor  or  platform  commands). 

-  A  low  level  communication  service  that  provides  data  flow 
between  the  processes  and  platforms.  Examples  of  data  are 
sensor  (detection)  reports  and  platform  or  sensor  com¬ 
mands.  It  primarily  transports  sensor  or  platform  related 
data  messages. 

-  A  high  level  communication  service  that  provides  data  flow 
between  the  processes  themselves.  Examples  of  data  are  sit¬ 
uation  reports  and  high  level  control  commands  to  sensor 
manager. 

-  One  or  more  man-machine-interfaces,  which  in  general 
have  access  to  all  data  stored  in  the  sensors/platforms, 
nodes  or  data  bases. 


-  -  I  ►  Control  (sensor  directives,  control  of  fusion  process 
which  is  not  depicted, ...) 


Figure  3  -  TA10  Demonstrator  Architecture 

(as  a  specific  instance  of  generic  architecture) 

The  data  fusion  nodes,  situation  assessment  node  and  the  sen¬ 
sor/platform  management  nodes  and  their  interaction  patterns 
(i.e.  processing  scheme)  make  up  a  specific  architecture  such 
as  the  TAD  core  (i.e.  demonstrator  architecture).  These  ele¬ 
ments  are  configured  into  a  specific  architecture,  processing 
the  data  from  a  specific  set  of  platforms  and  sensors.  A  specific 
instance  of  the  generic  architecture  for  the  TAD  can  be  found 
in  figure  3. 

The  distinction  between  low  level  and  high  level  communica¬ 
tion  services  is  somewhat  specific,  suggesting  the  existence  of 
a  more  generic  architecture.  However,  this  distinction  is  impor¬ 
tant,  because  it  introduces  the  difference  between  two  levels  of 
data  exchange,  which  also  enhances  the  construction  of  generic 
(invariant)  interfaces.  In  practice,  these  two  services  make  use 
of  the  same  underlying  network. 

The  reconfigurability  and  flexibility  with  respect  to  sensors, 
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platforms,  processing  scheme  and  application  domain  require 
generic  interfaces  of  the  architectural  elements,  and  a  common 
(internal)  structure  for  the  elements.  In  this  case,  a  specific 
architecture  with  specific  elements  is  nothing  more  than  an 
instance  of  the  generic  architecture  tailored  to  the  application 
with  respect  to  available  sensors  and  platforms,  application 
domain  and  selected  processing  scheme.  The  demonstrator 
architecture  is  an  instance  of  such  a  generic  architecture,  with  a 
fixed  fusion  network. 


1.2  DEMONSTRATOR  ARCHITECTURE 

The  TAD  core  is  based  on  a  network  of  processing  nodes 
(fusion  nodes,  assessment  node,  and  sensor/platform  manage¬ 
ment  nodes)  distributed  on  static  or  moving  platforms  con¬ 
nected  by  a  communication  network.  The  TAD  is  not  focused 
on  the  simulation  of  the  communication  network  but  shall  take 
into  account  the  existence  of  a  communication  network.  A  way 
to  do  so,  is  to  express  the  communication  requirements  of  the 
TAD  core  on  a  fully  networked  asynchronous  environment: 

a.  not  local  (distinct  locations), 

b.  dynamically  configurable  (availability  of  processing  nodes), 

c.  transport  protocol  independent  (diverse  and  evolving  com¬ 
munication  resources), 

d.  dynamically  link  configurable  (availability  of  communica¬ 
tion  resources). 

The  a)  and  c)  requirements  are  right  for  the  anticipated  future 
evolution  of  the  TAD  (distribution  of  processing  nodes  on  dis¬ 
tinct  hardware  resources  with  diverse  communicating  capabili¬ 
ties)  and  are  not  considered  as  demanding  requirements  for  the 
TAD  to  be  developed  in  the  project.  The  points  a),  c)  and  d)  are 
to  be  fulfilled  by  the  communication  network  services.  The  d) 
point  is  not  required  for  the  TAD.  The  TAD  uses  a  transport 
level  connection  oriented  reliable  byte  stream  service  for  the 
communication  between  processing  nodes. 

The  design  of  the  TA10  demonstrator  is  based  on  the  specific 
instance  of  the  generic  architecture  as  described  above.  This 
specific  demonstrator  architecture  defines  the  selected  sensor/ 
platform  configuration  and  the  processing  scheme  being  the 
TAD  core.  The  demonstrator  architecture  is  depicted  in  figures 
4  and  5. 
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Figure  4  -  Demonstrator  Architecture:  Fusion  nodes 


Figure  4  shows  the  data  flow  between  the  sensor  platforms  and 
fusion  nodes  as  a  result  of  the  processing  of  incoming  sensor 
data.  Figure  5  shows  the  flow  of  control  between  the  sensor 
management  nodes  and  sensor  platforms  as  a  result  of  the  Data 


Fusion  and  Situation  Assessment  process. 


J*  Management  data 
Fused  information 
Platform  |  1  Management  node 

cm  DF&SA  node 
I  I  Sensor 

Figure  5  -  Demonstrator  Architecture:  Sensor  management 

The  following  parameters  define  the  specific  TAD  architecture 
(see  also  list  of  abbreviations): 

-  Application  domain 

.  Battlefield  surveillance  (defined  by  scenario) 

-  Platforms 

.  Heliborne  Platform  (HP)  with  sensors  HESM  and  HMTI 
.  Ground  Based  Platform  1  (GBP])  with  sensor  GMTI] 

.  Ground  Based  Platform  2  (GBP2)  with  sensor  GMTI2 
.  Ground  Based  Platform  3  (GBP3)  with  GTV3  and  GIR3 
.  Ground  Based  Platform  4  (GBP4)  with  sensor  GIR4 
.  Ground  Based  Platform  5  (GBP5)  with  sensor  GTV5 
.  Drone  Platform  (DPi)  with  sensor  DIR, 

.  Drone  Platform  (DP2)  with  sensor  DTV2 

-  Fusion  nodes  (comprising  TAD  core  Data  fusion  function) 

.  HDFS  fusing  HMTI  and  HESM  (fusion  of  different 

types  of  sensors  on  the  same  platform) 

.  GDFSj  fusing  GMTI]  and  GTV5  (fusion  of  different 
types  of  sensors  on  different  platforms) 

.  GDFS2  fusing  GMTI2  and  GIR4  (fusion  of  different 
types  of  sensors  on  different  platforms) 

.  GDFS3  fusing  GTV3  and  GIR3  (fusion  of  different  types 
of  sensors  on  the  same  platform) 

.  GDFS:  This  composed  node  receives  the  fused  data 
coming  from  the  previous  nodes  (HDFS,  GDFS], 

GDFS2  and  GDFS3).  This  system  contains  three  hierar¬ 
chical  nodes: 

.  GGBDF  fusing  GDFS],  GDFS2  and  GDFS3.  This 
node  has  to  fuse  data  coming  from  the  three  ground 
based  data  fusion  nodes; 

.  GGBHDF  fusing  GGBDF  and  HDFS.  This  node  has 
to  fuse  data  coming  from  the  GGBDF  outputs  and  the 
Heliborne  Data  Fusion  System  (HDFS)  outputs; 

.  GDF  fusing  GGBHDF,  DIR]  and  DTV2.  This  node 
has  to  fuse  data  coming  from  the  GGBHDF  outputs 
with  the  information  extracted  from  the  data  collected 
by  the  drone(s). 

-  Situation  assessment  nodes  (comprising  TAD  core  Situation 
Assessment  function) 

-  Sensor  and  platform  management  nodes  (comprising  TAD 
core  Sensor/Platform  Management  function) 

.  GSMGT  taking  as  input  orders  from  operator,  (local) 
data  fusion  nodes  and  SA  and  prepare  orders  for 
LHSMGT,  LGSMGT],  LGSMGT2,  LGSMGT3  (control 
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of  DP[  and  DP2  is  also  supported  by  the  GSMGT). 
LHSMGT  taking  as  input  orders  from  GSMGT,  HDFS 
and  possibly  a  local  operator  (on  board),  and  prepare 
orders  for  HP  (i.e.  pilot,  flight  path),  HESM  and  HMTI. 

.  LGSMGT j  taking  as  input  orders  from  GSMGT,  GDFSi 
and  possibly  local  operator  and  prepare  orders  for  GBPj, 
GMTIj,  GBP5  and  GTV5. 

.  LGSMGT2  taking  as  input  orders  from  GSMGT,  GDFS2 
and  possibly  local  operator  and  prepare  orders  for  GBP2, 
GMTI2,  GBP4  and  GIR4. 

.  LGSMGT3  taking  as  input  orders  from  GSMGT,  GDFS3 
and  possibly  local  operator  and  prepare  orders  for  GBP3, 
GTV3,  and  GIR3. 

-  Man  Machine  Interface 

In  general  and  from  an  operational  point  of  view,  the  man 
machine  interface  which  addresses  the  interaction  needs  of 
one  or  more  operators  has  access  to  all  data  and  informa¬ 
tion,  whenever  useful.  Possibly,  MMI  parts  can  be  co¬ 
located  on  platforms  or  other  distant  places,  having  immedi¬ 
ate  access  to  the  local  data  and  providing  operator  control  to 
platform  and  sensors.  E.g.,  an  MMI  part  might  be  located 
on  the  helibome  platform  that  enables  the  local  operator  to 
examine  HMTI  and  HESM  data  and  the  results  from  fusion 
node  HDFS,  as  well  as  to  control  sensors  /  platform. 


2.  MAIN  FUNCTIONAL  COMPONENTS 

Apart  from  supporting  knowledge  and  databases  the  TAD  con¬ 
sists  mainly  of  one  Hardware  Configuration  Item  (HWCI)  and 
three  Computer  Software  Configuration  Items  (CSCI): 

-  ‘TAD  Workstation’  HWCI 

The  TAD  workstation  is  the  processing  resource  which  sup¬ 
ports  the  computing  and  operator  interaction  needs  of  TAD. 

-  Input  Simulator  CSCI 

The  CSCI  simulates  the  battlefield  and  sensors/platforms 
with  the  generation  of  sensor  reports  as  the  objective. 

-  Data  Fusion  and  Situation  Assessment  CSCI 

This  CSCI  implements  the  data  fusion  and  situation  assess¬ 
ment  nodes  with  as  main  functions  to  provide  the  operator  a 
synthetic  picture  of  the  battlefield  and  guidance  to  Sensor/ 
Platform  Management  functions  through  requests. 

-  Sensor/Platform  Management  CSCI 

This  CSCI  implements  the  sensor/platform  management 
nodes.  Their  main  function  is  to  provide  steering  commands 
to  sensors/platforms  based  on  data  fusion,  situation  assess¬ 
ment  or  operator  requests  in  order  to  have  an  efficient  and 
effective  deployment  and  use  of  these  sensors/platforms. 


2.1  SCENARIO  AND  SIMULATION 
OPERATIONAL  AREA 

The  area  where  the  scenario  will  take  place  is  a  20  x  20  km 
square  area.  This  terrain  is  digitized  taking  into  account  (both 


during  simulation  and  during  the  execution  of  the  Data  Fusion, 
Situation  Assessment  and  Sensor/Platform  Management  proc¬ 
esses)  relief  (hills),  civil  engineering,  and  vegetation  aspects, 
in  order  to  address  intervisibility  and  background  problems. 

ENEMY  AND  NEUTRAL  TARGETS 

The  agreed  represented  scenario  includes  100  potential  targets: 

-  50  civilian  ones  which  can  move  randomly  on  road  network 

-  20  military  vehicles  (cars,  light  vehicles,  tanks)  which  move 
in  accordance  with  military  doctrine.  In  opposition  to  cars, 
tanks  are  not  limited  to  roads.  The  scenario  will  only  let 
them  leave  the  roads  if  in  accordance  with  military  doctrine. 

-  30  military  units  (Vehicles,  troops)  moving  in  accordance 
with  military  doctrine. 


OWN  SENSORS 

The  sensors  are  all  friends.  The  enemy  sensors  are  considered 
only  as  emitters.  The  friendly  sensors  include: 

-  1  single  airborne  ESM  (bearing,  sensor  characteristics)  + 
Long  range  MTI  radar  (range,  bearing,  doppler), 

-  2  fixed  ground  medium  range  MTI  radar, 

-  1  fixed  ground  IR  +  TV  camera  (bearing,  bearing  radial 
velocity,  elevation,  elevation  radial  velocity), 

-  1  fixed  ground  IR  camera, 

-  1  fixed  ground  TV  camera, 

-  1  drone  equipped  with  IR  camera, 

-  1  drone  equipped  with  TV  camera. 

All  landbased  sensors  are  fixed  and  distributed  along  the  FLOT 
(Forward  Limit  of  Own  Troops)  in  order  to  assure  a  coverage 
overlapping  (see  figure  7).  As  there  is  no  battle,  FLOT  is  used 
instead  of  FEB  A  (Forward  Edge  of  Battle  Area). 

The  drone  platforms  are  sent  at  the  appropriate  time  for  a  30  to 
120  minutes  mission.  That  mission  is  planned  by  the  system 
and  validated  by  the  operator,  as  deducted  from  the  data  fusion 
system  and  the  current  assessed  situation  especially  about  the 
areas  with  low  quality  information  and/or  potential  risks. 
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The  airborne  MTI  radar  is  installed  on  a  helicopter.  The  heli¬ 
copter  flight  path  has  the  form  of  a  race  course  (60x10  km)  at  a 
40  km  stand-off  distance  from  the  frontier.  It  delivers  three  glo¬ 
bal  radar  pictures  of  the  area  per  track. 

Another  important  point  is  to  distinguish  the  theoretical  detec¬ 
tion  range  of  the  sensors  and  their  practical  detection  range. 
The  theoretical  range  is  the  range  obtained  in  the  best  condi¬ 
tions  for  the  meteo  and  without  any  masking  nor  by  the  relief 
nor  by  the  terrain  coverage.The  practical  range  will  be  limited 
more  or  less  by  the  meteorological  conditions,  and  mainly  by 
the  terrain  relief  and  coverage.  The  ranges  shown  in  the  sce¬ 
nario  figure  above  are  the  average  practical  ranges  in  opera¬ 
tional  conditions. 

SIMULATION  REMARKS 

1 .  The  battlefield  is  chosen  in  the  south  of  France  in  a  zone 
with  moderate  relief  and  few  forests,  so  the  masking  effects 
will  not  dramatically  decrease  data  fusion  possibilities. 

2.  Due  to  microrelief,  the  real  detection,  even  in  the  detection 
range,  will  not  be  equal  to  100%. 

3.  The  battlefield  includes  several  possible  military  objectives 
to  get  an  anticipated  estimation  of  the  situation  assessment. 

4.  The  enemy  is  assumed  to  be  limited  to  the  battlefield  area 
and  does  not  to  have  interactions  with  other  troops  outside. 
It  will  be  thus  a  punctual  action  near  the  frontier  between 
two  stabilized  lines. 

5.  The  operator  is  assumed  to  know  what  the  potential  military 
objectives  in  the  observed  zone  are. 

6.  At  time  t=0  the  operator  is  assumed  to  have  no  other  infor¬ 
mation  then: 

-  the  battlefield  map, 

-  the  potential  military  objectives  in  the  observed  zone, 

-  the  enemy  military  doctrine, 

-  the  information  on  his  own  sensors. 

7.  The  enemy  does  not  react  to  the  observation. 

8.  During  the  demonstration  there  is  no  reaction  from  friendly 
troops. 

9.  For  the  operator  (or  observant)  the  starting  state  of  the 
external  world  is  totally  unknown.  After  15  minutes  of  evo¬ 
lution  of  civil  targets,  new  targets  of  the  military  type  can  be 
injected  into  the  field: 

-  Military  vehicles  up  to  1  per  minute,  with  max.  of  20. 


-  Military  units  up  to  1  each  5  minutes,  with  max.  of  30. 
These  injected  targets  constitute  the  initial  total  of  50  mili¬ 
tary  objects  (vehicles  or  units).  Injection  of  targets  will  only 
happen  at  the  borders  of  the  20  x  20  km  battlefield. 


2.2  KNOWLEDGE  AND  DATABASES 

This  section  describes  the  data  and  knowledge  in  the  TAD.  The 
data  and  knowledge  needed  for  the  TAD  consists  of  world 
model  data/knowledge.  The  world  model  contains  the  data  and 
knowledge  that  describes  the  battlefield.  An  object-oriented 
approach  is  followed  by  defining  world  model  objects  in  a  way 
that  has  an  equivalent  on  the  battlefield.  Each  object  encapsu¬ 
lates  three  types  of  data  or  knowledge: 

1.  Static  data:  static  attributes  of  objects  of  the  world.  Static 
data  only  consists  of  data  that  does  not  change  during  a 
demonstration,  e.g.  the  typical  size  of  a  civil  target. 

2.  Dynamic  data:  dynamic  attributes  of  objects  of  the  world. 
Dynamic  data  change  during  a  demonstration  and  are 
defined  as  a  function  of  time,  e.g.  the  position  of  a  military 
target. 

3.  Behaviour  knowledge:  knowledge  that  describes  the  behav¬ 
iour  of  objects  in  the  world,  e.g.  how  the  terrain  conditions 
influence  the  speed  of  a  military  target. 

The  Input  Simulator  uses  an  instance  of  the  world  model  to 
perform  a  simulation  according  to  a  specific  scenario.  The 
TAD  core  uses  and  maintains  one  or  more  instances  of  the 
world  model  with  as  goal  to  reconstruct  the  simulated  scenario 
on  basis  of  unreliable  sensor  reports.  In  this  case,  the  objects  in 
the  world  model  are  track  hypotheses.  The  different  instances 
of  the  world  model  are  implemented  in  each  CSCI  or  Compu¬ 
ter  Software  Component  (CSC),  according  to  its  specific  needs. 
Several  CSCs  might  share  the  same  instance. 


2.2.1  WORLD  MODEL 

The  world  model  consists  of  track  hypotheses.  Each  hypothesis 
contains  data  and/or  knowledge  about  the  natural  object  as 
found  in  the  battlefield.  It  is  assumed  that  this  set  of  objects 
contains  the  basic  data  and/or  knowledge  for  constructing  the 
world  model.  The  following  objects  are  identified  in  the  world 
model:  Scenario,  Battlefield,  Atmosphere,  Terrain,  Entity,  Sen¬ 
sor,  Platform,  Military  Target,  Military  Unit,  and  Civil  Target. 

Figure  8  describes  the  hierarchical  relationships  in  terms  of 
‘HAS  A’  and  ‘IS  A’  relations.  The  relation  ‘HAS  A’  indicates 
that  an  object  is  composed  of  its  parts  (which  are  again 
objects),  e.g.  the  object  ‘battlefield’  consists  of  the  objects 
‘atmosphere’,  ‘terrain’,  and  a  number  of  ‘entities’.  The  relation 
‘IS  A’  indicates  a  subtype/supertype  relationship, e.g.  the 
objects  ‘platform’,  ‘sensor’,  ‘military  unit’,  ‘military  target’ 
and  ‘civil  target’  are  all  a  subtype  of  the  object  ‘entity’ .  This 
implies  that  certain  attributes  of  ‘entity’,  such  as  ‘location  of 
the  object’,  are  inherited  by  the  subtypes. 

The  object  ‘Battlefield’  is  regarded  as  having  an  atmosphere 
(e.g.  weather  conditions),  terrain  characteristics  and  a  set  of 
entities,  such  as  military  targets  and  sensors.  A  ‘scenario’  is 
regarded  as  multiple  instances  of  ‘battlefield’  as  a  function  of 
time  in  order  to  capture  the  dynamic  aspect  of  the  battlefield 
(note  that  each  object  will  have  a  time  line,  i.e.  history,  present 
and  possibly  future).  The  ‘HAS  A’  relation  between  ‘platform’ 
and  ‘sensor’  indicates  that  a  platform  consists  of  one  or  more 
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sensors.  The  ‘HAS  A’  relation  of  ‘military  unit’  to  itself  mod¬ 
els  the  fact  that  a  military  unit  can  be  composed  of  subunits 
(e.g.  a  battalion  is  composed  of  companies),  where  the  smallest 
military  unit  (i.e.  platoon)  is  composed  of  ‘military  targets’ 
such  as  tanks  and  personnel. 


2.2.2  OBJECT  IDENTIFICATION 

For  each  object  the  name,  description  and  examples  of  the 
static  and  dynamic  data  is  given.  The  behaviour  knowledge 
associated  with  each  object  is  only  named. 


Name 

Description 

Static  data 

Dynamic 

data 

Behaviour 

Scenario 

The  Scenario  object  specifics  the 
battlefield  as  a  function  of  time.  The 
relation  of  objects  with  time  is 
necessary  in  order  to  capture  the 
dynamic  aspects  of  the  battlefield.  This 
means  that  objects  and  their  attributes 
are  defined  as  function  ofiimc. 

Time  reference 

Time 

Incorporated  in 
the  behaviour  of 
each  object  as  a 
function  of  time. 

Battlefield 

The  Battlefield  object  specifies  the  real 
world  at  an  instant  of  time.  Main  aspect 
to  identify  this  object  is  to  represent  a 
common  frame  of  reference  for  the 
objects  that  are  being  modelled: 
atmosphere,  and  terrain,  and  entity 

Coordination 

grid 

Defined  by 
the  objects: 
Atmosphere, 
Terrain  and 
Entity 

Defined  by  the 
objects: 
Atmosphere, 
Terrain  and 

Entity 

Atmosphere 

The  Atmosphere  object  specifies 
atmospheric  aspecLs  of  the  battlefield. 
This  object  consists  of  objects,  such  as: 
daylight  object  and  weather  object 

Humidity, 

temperature. 

Location 

Environment 

behaviour 

Terrain 

The  Terrain  object  specifies 
information  about  the  battlefield 
terrain.  Two  types  of  terrain  objects 
exist:  natural  objects,  such  as 
mountains  and  lakes  and  man-made 
objects,  such  as  roads,  bridges,  and 
canals. 

Location  of  lakes 

Bridges  might 
be  destroyed 

Environment 

behaviour 

Entity 

The  Entity  object  is  an  abstract 
representation  of  the  objects:  Platform, 
Sensor,  Civil  target,  Military  target, 
and  Military  unit.  Entity  object 
provides  generic  attributes  which  are 
inherited  by  its  subtypes.  For  example, 
typical  the  attributes  are:  speed, 
location,  and  size. 

Size  and 
maximum  speed 

Speed  and 
location 

Defined  by  the 
objects:  Platform, 
Sensor,  Civil 
Target,  Military 
Target,  and 
Military  Unit. 

Platform 

The  Platform  object  specifies 
characteristics  of  a  platform. 

Set  of  sensors 
located  on  a 
platform 

Platform 

altitude 

Sensor/Platform 

Behaviour 

Sensor 

The  Sensor  object  specifies  sensor 
characteristics. 

Frequency 
(maximum  and 
minimum) 

Sensor 

altitude 

Sensor/Platform 

Behaviour 

Civil  Target 

The  Civil  target  object  specifies  the 
characteristics  of  targets  such  as  buses 
and  cars. 

Number  of 
civil  vehicles 
involved 

Civil  Target 
Behaviour 

Military 

Target 

The  Military  target  object  specifies 
military  enemy  targets. 

Armed 

Tlircat 

indication 

Military  target 
Behaviour 

Military  Unit 

Military  Unit  object  specifies  which 
targets  form  a  unit  and  it  also  specifics 
the  characteristics  of  this  unit. 

Typical  structure 
(composition) 

Structure 

Unit  Behaviour 

2.2.3  BEHAVIOUR  KNOWLEDGE 

The  behaviour  knowledge,  as  listed  in  the  previous  section  for 
each  object,  consists  of  knowledge  of  the  behaviour  of  single 
object  and  possibly  in  relation  with  other  objects.  The  knowl¬ 
edge  capturing  the  behaviour  of  the  world  model  objects  is 
identified  in  the  following  five  behaviour  items: 


Name 

Description 

Involved  objects 

Environment 

Behaviour 

Knowledge  describing  the  interaction  between  the  atmosphere 
and  the  terrain.  E.g.  ihe  rain  makes  the  terrain  muddy. 

Atmosphere,  Terrain 

Sensor/ 

Platform 

Behaviour 

Knowledge  describing  the  behaviour  of  sensors/platforms  with 
respect  to  the  type  of  sensor,  tire  current  atmosphere,  terrain,  and 
targets  to  be  detected,  tracked  and  identified.  E.g.  the  weather 
influences  the  sensor  performance. 

Atmosphere,  Terrain, 
Sensor/Platform,  Military 
Target,  Civil  Target 

Military  Target 
Behaviour 

Knowledge  describing  the  behaviour  of  military  targets  with 
respect  to  terrain,  atmosphere,  type  of  military  target,  and  military 
units  (command  and  control).  E.g.  the  terrain  influences  the  speed 
of  a  military  target. 

Terrain,  Atmosphere, 
Military  Target,  Military 
Unit 

Civil  Target 
Behaviour 

Knowledge  describing  the  behaviour  of  civil  targets  with  respect 
to  terrain,  atmosphere,  and  type  of  civil  target.  E.g.  the  terrain 
influences  the  speed  of  a  civil  target. 

Terrain.  Atmosphere, 
Military  Target 

Unit  Behaviour 

Knowledge  describing  the  behaviour  of  military  units  (ORB  AT: 
Order  of  Battle)  with  respect  to  terrain,  atmosphere,  military  units 
(structure  and  targets).  E.g.  a  muddy  terrain  slows  down  the 
movement  of  troops. 

Terrain,  Atmosphere, 
Military  Target,  Military 
Unit 

2.3  DATA  FUSION  AND  SITUATION  ASSESSMENT 

Figure  4  shows  that  the  TA10  data  fusion  occurs  in  a  number  of 
different  distributed  data  fusion  nodes.  In  the  different  fusion 
and  assessment  nodes  sensor  information  or  fused  information 
is  taken  together  to  produce  a  more  complete  overview  of  the 
battlefield.  The  information  is  based  on  plots,  tracks  or  already 
fused  information  [1][2][3][5]. 


P,  R.  R,  vehicle  types/numbers 

MT1:| 


MT^ 


P,  P,  E,  E,  vehicle  types/numbers 
TV/1R: 


R:L 


Data  Alignment 


Association  /  Correlation 

1 


Positional  fusion  (filtering) 

t 


Identity  fusion 

~r~ 


Hypotheses  Reduction  | 


Spatial  Grouping 

I 


situation  overview 


Figure  9  -  Overview  of  fusion  and  assessment  process 


N.B.:  In  this  paragraph  the  various  items  of  the  fusion  process 
will  be  discussed.  Although  various  combinations  of  sensors, 
each  with  their  own  peculiarities,  are  used,  we  will  mainly 
limit  ourselves  to  the  example  of  the  fusion  node  GDFSj,  in 
which  MTI  and  TV  data  is  fused. 


2.3.1  DATA  ALIGNMENT 

Data  Alignment  can  be  split  in  alignment  in  time  (Temporal 
alignment)  and  in  space  (Spatial  alignment).  The  result  of 
Temporal  alignment  is  a  set  of  measurements  in  the  same  time 
reference.  The  measurements  from  each  sensor  are  expressed 
with  respect  to  the  sensor  position.  To  avoid  annoying  covari¬ 
ance  terms  when  a  measurement  is  correlated  with  a  track,  the 
predicted  position  for  this  track  is  expressed  in  the  coordinate 
system  of  the  corresponding  sensor.  In  general,  measurements 
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from  two  different  sensors  can  be  combined  as  desired,  if  it  is 
specified  how  to  translate  measurements  from  the  two  sensors 
in  their  own  coordinate  system,  to  and  from  a  global  coordinate 
system.  In  practice,  more  efficient  methods  will  be  used.  E.g., 
in  the  current  situation  measurements  will  be  translated  from 
the  TV  coordinate  system  to  the  MTI  coordinate  system  and 
visa  versa.  Only  at  the  end  of  the  fusion  process,  when  the 
results  are  transferred,  a  translation  to  the  global  coordinate 
system  will  be  made. 

E.g.,  the  MTI  radar  gives  range  and  azimuth  information  of 
objects  detected  in  a  specific  elevation  domain.  This  is  for  each 
detected  object  an  arc  with  as  centre  the  position  of  the  MTI 
radar.  The  TV  camera  and  IR  camera  give  azimuth  and  eleva¬ 
tion  information  of  objects  detected  within  a  specified  range 
domain.  This  is  for  each  detected  object  part  of  a  straight  line 
starting  from  the  position  of  the  camera. 


2.3.2  ASSOCIATION  AND  CORRELATION 

Association  and  correlation  involve  sorting  or  correlating 
observations  from  multiple  sensors  into  data  sets,  with  each 
data  set  representing  data  related  to  a  single  distinct  entity. 


The  decision  which  one,  if  any,  to  assign  to  track  1  can  be 
postponed.  MHT  supports  8  hypotheses,  among  which  A 
and  B,  and  evaluates  them  in  parallel.  Two  predicted  posi¬ 
tions  and  corresponding  gates  are  computed: 


Track  1 


Hypothesis  A 

pi  / 

VrC't 


p7  * 

Hypothesis  B  \ 


) 


When  the  next  data  scan  is  received,  one  plot  turns  out  to 
fall  fairly  centrally  in  the  gate  of  the  Hypothesis  B  track, 
and  on  the  outskirts  of  the  Hypothesis  A  track.  Depending 
on  the  setup,  the  MHT  algorithm  may  conclude  that  both 
hypothesis  will  be  supported  further,  or  accept  Hypothesis 
B.  Accepting  Hypothesis  B  can  for  instance  occur  when 
Hypothesis  A  has  become  so  unlikely  that  it  is  removed. 


Now  the  data  alignment  processes  have  established  that  meas¬ 
urements  can  be  compared  with  each  other,  it  is  possible  to 
determine  which  observations  may  have  been  caused  by  the 
same  object.  In  the  Kalman  filtering  used  in  Positional  fusion, 
for  each  combination  of  a  sensor  and  a  track,  a  correlation  win¬ 
dow  is  defined.  Only  measurements  of  the  sensor  in  this  corre¬ 
lation  window,  are  correlated  with  the  track.  The  current  fusion 
step  has  to  do  association  and  correlation,  thus,  has  to  create 
this  correlation  window.  Among  other  approaches,  in  TA10 
Multiple  Hypothesis  Tracking  is  applied  [4], 

The  aim  of  Multiple  Hypothesis  Tracking  is  to  postpone  the 
final  decision  on  difficult  association  situations  until  the  receipt 
of  more  information.  A  number  of  candidate  hypothesis  will  be 
generated  and  evaluated  later  on,  as  more  data  is  received.  The 
method  is  recursive,  such  that  data  sets  only  need  to  be  proc¬ 
essed  on  receival.  Central  to  the  MHT  approach  is  the  forma¬ 
tion  of  a  hypothesis  tree.  Each  hypothesis  in  the  tree  contains  a 
different  series  of  plausible  ways  to  partition  information  into 
tracks  and  false  alarms.  Each  time  a  new  set  of  observation 
data  arrives,  the  tree  is  maintained  by  individually  evaluating 
every  existing  hypothesis.  A  track  occurring  in  one  of  more 
hypotheses  is  evaluated  until  it  is  eliminated  (if  the  probability 
of  it  being  a  real  track  is  too  low),  or  turned  into  a  definite  track 
(if  the  probability  of  it  being  a  real  track  is  high  enough).  If  a 
hypothesis  contains  a  hypothesised  track  that  is  eliminated,  this 
hypothesis  cannot  be  true  thus  it  is  eliminated  too.  If  a  hypoth¬ 
esis  contains  a  hypotheses  track  that  is  turned  into  a  definite 
one,  this  part  of  the  hypothesis  is  true,  and  only  the  remaining 
part  has  to  be  evaluated  and  becomes  a  ‘new’  hypothesis.  A 
simple  example  will  clarify  the  MHT  principle: 

The  line  connecting  four  plots  below  represents  the  move¬ 
ment  of  track  1 .  The  V  is  the  predicted  position  based  on 
these  four  plots,  and  the  ellipse  shown  is  the  correlation 
window  belonging  to  this  predicted  position.  Two  received 
plots  turn  out  to  fall  within  this  window: 


Track  1 


Hypothesis  A 

+ 


p? 

Hypothesis  B  i 
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Below  the  corresponding  MHT  tree  is  depicted  for  the  situ¬ 
ation  as  above  where  two  plots,  PI  and  P2,  are  received  of 
which  at  most  one  can  be  assigned  to  the  existing  track. 
The  tree  reads  from  left  to  right  and  is  structured  in  a  series 
of  layers  that  represent  alternative  assignments  of  the  plots. 
The  notation  is: 

FA  :  observation  taken  to  be  a  False  Alarm; 

NTi  :  observation  initiates  the  New  Track  nr  i,  hereafter 
referred  to  as  Ti; 

T /(O  :  observation  associated  with  existing  Track  i, 
which  results  in  the  extended  track  Tj. 

HI 
H2 

H3 
H4 

HS 
H6 

H7 

H8 
P2 


PI 


The  track  renumbering  T2(l)  caused  by  the  assignment  of 
plot  1  to  the  existing  track  T1  is  done  to  preserve  the  exist¬ 
ence  of  T1  which  may  be  needed  later,  so  that  the  (hypoth¬ 
esised)  track  2  contains  all  assignments  of  track  1  with  the 
additional  plot  PI.  Even  in  the  simple  example,  identical 
tracks  appear  in  more  than  one  hypothesis.  For  instance, 
T4  corresponding  to  Hypothesis  B  shows  up  in  both  H2 
and  H7  (see  combining  paragraph  2.3.5). 


2.3.3  POSITIONAL  FUSION 

In  the  TA10  positional  fusion  process,  multiple  observations  of 
positional  data  of  one  object  are  combined  in  order  to  deter¬ 
mine  an  estimate  of  the  state  vector  using  (extended)  Kalman 
filtering  [7],  Note  that  the  data  includes  all  available  data  of  the 
state  vector  of  the  object,  thus  not  only  the  position,  but  also 
for  instance  the  (Doppler)  velocity.  This  information  in  combi¬ 
nation  with  the  geographical  information  is  used  to  select 
between  different  object  behavioural  models. 


Optronic  2 


Figure  10  -  Example  of  obstruction  in  terrain 

Besides  the  information  on  the  objects  also  information  from 
the  environment  is  taken  into  account  (figure  10).  Assume  e.g. 
that  two  targets  have  been  observed  by  both  the  radar  and  the 
optronic  sensor  and  that  the  Optronic  sensor  (without  range 
information)  view  in  the  directions  of  measurement  Optronic  1 
is  hampered  by  an  obstacle,  like  a  hill.  In  principle  there  are 
four  possible  positions  for  commonly  detected  targets  (candi¬ 
date  1  -  4).  Due  to  the  hampered  Optronic  sensor  view  candi¬ 
date  3  is  a  less  likely  object  position  than  candidate  4  is  in  the 
Optronic  1  direction.  If  therefore  candidate  4  is  selected  in  the 
Optronic  1  direction,  the  only  candidate  for  another  commonly 
detected  target  is  candidate  2. 

If  in  a  certain  area  the  probability  of  the  occurrence  of  clutter  is 
high,  the  probability  that  a  measurement  in  this  area  corre¬ 
sponds  to  a  false  alarm  should  be  increased.  In  other  words, 
this  candidate  is  less  likely  than  another  candidate  in  another 
area,  which  regarding  other  aspects  was  equally  likely.  It  has  to 
be  mentioned  that  whether  or  not  a  certain  position  is  likely  to 
be  an  object  position,  is  different  for  the  various  kinds  of 
objects.  Thus  the  probability  of  a  certain  position  being  a  real 
object  position  is,  among  others,  dependent  on  the  kind  of 
object  the  detection  is,  or  is  expected  to  be.  The  probability  that 
a  tank  is  measured  in  a  river  is  considerably  lower  than  the 
probability  that  an  amphibious  vehicle  is. 

Furthermore,  in  the  filtering  process  of  some  fusion  nodes  in 
TA10  the  object  behaviour  related  to  the  geographical  and 
meteorological  environment  is  taken  into  account.  E.g.  the  ter¬ 
rain  accessibility  information: 

1.  If  a  military  vehicle  is  travelling  on  a  road,  it  is  more  likely 
that  it  will  continue  to  travel  on  the  road,  then  that  it  will 
leave  the  road  and  enter  bushes,  even  though  it  is  able  to 
enter  the  bushes. 


(A)  (B)  (C) 


The  position  predicted  by  the  Kalman  filter  is  usually  taken 
as  the  (one  and  only)  expected  next  update  position  of  the 
track.  In  most  situations,  like  the  one  depicted  in  example 
(A)  above,  this  is  a  logical  and  sane  decision.  However,  it 
can  happen  that  this  movement  involves  a  change  to  a  less 
preferred  terrain  while  this  could  be  avoided.  For  example, 
in  situation  (B),  the  Kalman  filter  prediction  is  that  the 
object  moves  into  the  river,  while  without  further  informa¬ 
tion  it  is  more  sensible  that  the  object  turns  somewhat  to  the 
right  and  follows  the  border  of  the  river.  Therefore  one 
might  want  to  adopt  a  rule  like: 

’If  the  filter  predicts  the  vehicle  to  move  into  terrain 
which  is  at  least  as  good  as  the  terrain  the  vehicle  is  cur¬ 
rently  moving  on,  assume  that  the  vehicle  is  not  going  to 
manoeuvre  to  change  its  direction  of  movememt.  If  the 
terrain  corresponding  with  the  filter  prediction  is  worse 
then  the  terrain  the  vehicle  is  currently  moving  on  while 
this  can  be  avoided,  assume  that  the  vehicle  is  possibly 
going  to  manoeuvre  or  is  manoeuvring  and  account  for  a 
non-negligable  (constant)  vehicle  acceleration  in  the  fil¬ 
tering  process.  ’ 

The  first  rule  implies  that  when  a  vehicle  is  approaching  a 
road  junction,  as  in  example  (C),  the  probability  of  going 
straight  on  is  the  same  as  when  there  would  not  have  been 
roads  to  the  left  and/or  right.  This  kind  of  behaviour  is 
desired  when  a  vehicle  is  for  example  moving  on  grass  or 
through  bushes,  but  not  when  it  is  moving  on  a  road.  There¬ 
fore  the  next  rule  is  added: 

’If  the  vehicle  is  moving  on  a  road  and  is  approaching  a 
road  junction,  assume  that  the  vehicle  is  possibly  going  to 
manoeuvre  or  is  manoeuvring  and  account  for  a  non-neg¬ 
ligable  (constant)  vehicle  acceleration  in  the  filtering 
process.  ’ 

2.  The  atmospherical  and  meteorological  elements  are  divided 
in  day/night  conditions  (either  day  or  night),  and  weather 
conditions  (clear  weather,  haze,  light  fog,  dense  fog,  small 
rain,  dense  rain).  For  each  of  the  possibilities  an  ‘atmos¬ 
pheric  and  meteorological  dependent’  template  will  be 
made  which  is  also  input  for  the  filtering  process.  Each  such 
template  defines  the  coverage  area  of  all  (relevant)  sensors 
for  the  atmospheric  and  meteorological  situation  in  the 
given  terrain.  It  has  to  be  decided  which  of  the  classes  is 
most  appropriate  for  the  current  atmospheric  and  meteoro¬ 
logical  situation,  and  the  corresponding  template  is  defined 
to  be  the  actual  one.  The  momentary  valid  template  can 
change  while  the  scenario  is  running,  for  example  when  it 
starts  or  stops  raining. 

The  atmospheric  and  meteorological  conditions  do  also 
have  influence  on  the  amount  of  clutter  that  is  present. 
When  the  amount  of  clutter  increases,  the  density  of  false 
targets  measured  by  a  radar  increases.  For  each  of  the  situa¬ 
tions  for  which  a  sensor  coverage  area  template  is  made, 
also  a  clutter  map  can  be  deduced  (or  measured).  This  has 
to  be  done  for  each  sensor. 
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2.3.4  IDENTITY  FUSION 

Identity  fusion  is  the  process  in  which  multiple  observations  of 
identity  data  are  combined  to  determine  a  decision  level  iden¬ 
tity  declaration.  In  this  stage  of  the  fusion  process,  the  simple 
rule  will  be  adopted  that  MTI  radar  identity  information  (e.g. 
fixed  wing,  or  helicopter)  will  only  be  used  if  it  is  the  only 
identification  information  available  for  a  certain  object,  other¬ 
wise  TV  camera  identity  information  will  be  used. 

Since  the  quality  of  TV  camera  identity  information  is  consid¬ 
erably  higher  than  that  of  the  MTI  radar,  only  using  TV  camera 
identity  information  for  the  fusion  result  is  good  enough  at  this 
level.  Later  on,  in  the  assessment  part  of  the  process  the  vari¬ 
ous  kinds  of  identity  information  that  can  be  obtained  from 
various  sources  (i.e.  not  only  sensors)  will  be  combined  in  a 
more  advanced  way.  The  use  of  templates  is  one  of  the  meth¬ 
ods  that  will  be  used. 

2.3.5  HYPOTHESIS  REDUCTION 

The  result  of  different  processes  is  a  number  of  possible  world 
model  hypotheses.  To  prevent  a  combinatorial  explosion  in  the 
number  of  hypotheses,  it  is  necessary  to  take  measures  to 
reduce  the  number  of  hypotheses,  which  is  known  as  hypothe¬ 
ses  reduction.  Four  techniques  are  used  to  manage  the  number 
of  hypotheses: 

1.  Windowing 

This  is  done  using  the  Kalman  filtering  information;  only 
measurements  that  are  within  a  gate  around  the  expected 
update  position  are  considered  for  correlation. 

2.  Clustering 

Observations  in  the  area  of  interest  are  divided  in  clusters, 
such  that  the  objects  in  distinct  clusters  do  not  interact  with 
each  other.  Each  cluster  will  subsequently  be  treated  as  an 
independent  area. 

3.  Pruning 

The  number  of  existing  hypotheses  is  unlimited  at  the 
moment.  In  the  step  4  it  will  be  investigated  whether 
hypotheses  can  be  combined.  For  feasibility  reasons  we 
would  like  to  have  an  upper  bound  on  the  number  of 
hypotheses.  This  is  achieved  by  simple  pruning  based  upon 
a  combination  of  breath  control  (maximum  number  of 
hypotheses)  and  adaptive  pruning  (when  the  sum  of  the 
probabilities  from  the  best  hypotheses  exceeds  a  certain 
threshold,  the  remaining  lower  probability  hypotheses  are 
being  pruned). 

4.  Combining 

It  is  investigated  whether  two  or  more  of  the  hypotheses 
that  survived  the  previous  step  have  enough  similarities  so 
they  can  be  combined. 

2.3.6  SPATIAL  GROUPING 

An  unit  (e.g.  company)  which  carries  out  an  operational  task 
(attack,  moving  in  column)  consists  of  vehicles  which  move  in 
close  proximity  and  have  similar  behaviour.  The  numerical 
processes  produce  a  situation  overview  containing  object 
tracks.  Tracks  which  are  close  together  and  have  a  similar 
behaviour,  have  a  (rather)  high  probability  to  belong  to  the 
same  unit.  Using  the  information  in  the  situation  overview, 
spatial  groups  are  formed. 

A  spatial  group  is  defined  as  a  number  of  vehicles  which  are 
related  because  of  their  close  proximity  and  similar  behaviour. 


When  the  situation  overview  contains  tracks  indicating  an 
attacking  hostile  unit,  a  spatial  group  is  formed  by  tracks 
within  a  certain  maximum  distance  from  the  centroid  of  the 
spatial  group  and  which  are  moving  in  the  same  direction  with 
similar  velocity.  The  required  radius  for  the  circle  to  approxi¬ 
mate  the  area  which  an  unit  covers  depends  on  the  selected 
smallest  unit  size  which  is  required  to  be  seen  on  the  situation 
overview.  In  the  case  that  the  situation  overview  indicates  units 
moving  in  column,  a  spatial  group  is  formed  by  tracks  within  a 
rectangle  positioned  around  the  centroid  of  the  spatial  group 
and  which  are  moving  in  the  same  direction  with  similar  veloc¬ 
ity.  The  size  of  the  rectangle  is  determined  by  the  smallest  unit 
size  which  is  required  to  be  seen  on  the  situation  overview. 


2.3.7  SITUATION  ASSESSMENT 

In  general  the  goal  of  Situation  Assessment  is  to  produce  com¬ 
bat  intelligence  concerning  the  enemy  operating  in  the  area  of 
interest.  Combat  intelligence  provides  an  understanding  of  the 
following  aspects: 

1 .  The  composition  of  the  enemy 

This  describes  the  type  and  size  of  the  enemy  units  and  their 
co-operation  in  combat  situations,  and  furthermore  the  iden¬ 
tification  of  the  observed  enemy  units. 

2.  The  disposition  of  the  enemy 

This  indicates  where  and  how  the  enemy  has  lined  up  its 
units  and  installations.  Together  with  the  composition  this 
can  reveal  the  (tactical)  grouping  of  the  enemy. 

3.  The  combat  effectiveness  of  the  enemy 

This  is  the  expression  of  the  ability  of  a  unit  to  pursue  a 
combat  activity  in  terms  of  the  quality  and  the  quantity  of 
the  available  personnel  and  equipment. 

4.  The  activities  of  the  enemy 

This  describes  conclusions  concerning  the  activities  of  the 
enemy  based  on  enemy  movements,  establishment  of 
assembly  areas,  the  transportation  of  supplies  etc.,  and  the 
comparison  of  current  and  previous  situation.  Activities  are 
compared  with  what  is  known  of  the  doctrinal  behaviour  of 
the  enemy. 

5.  The  peculiarities  and  limitations  of  the  enemy. 

This  aspect  denotes  the  strong  and  the  weak  elements  of  the 
enemy,  that  should  get  extra  attention  or  should  be 
exploited.  It  also  describes  features  such  as  air  support,  spe¬ 
cial  forces,  deviations  from  doctrine  etc. 

In  the  TA10  project,  Situation  Assessment  is  aimed  at  uncover¬ 
ing  composition,  disposition,  and  activities  of  the  enemy  and  at 
estimating  enemy  capabilities  and  intentions.  Input  to  the  Situ¬ 
ation  Assessment  process  will  be  reports  produced  by  the  Data 
Fusion  process  and  information  of  possible  enemy  objectives. 
Enemy  objectives  involved  in  the  TA10  project  will  be  inten¬ 
tions  to  occupy  a  physical  object  or  target  (e.g.  an  airfield  or 
ammunition  storage)  in  the  area  of  interest. 

As  mentioned  before,  the  ultimate  goal  of  Situation  Assess¬ 
ment  is  to  establish  enemy  intention,  intention  in  this  project 
being  related  to  the  occupation  of  a  geographical  target  (objec¬ 
tive).  In  order  to  do  so.  Situation  Assessment  will  establish 
what  units  are  active  in  the  area  of  interest,  estimate  current 
activities  and  predict  future  activities  in  direct  relation  to  the 
objective.  The  tasks  that  are  performed  in  this  process  are  the 
following: 

L  Unit  Classification 

This  task  deals  with  the  classification  of  groups  reported  by 


1-10 


the  Data  Fusion  process,  i.e.  determining  the  composition  of 
the  enemy.  The  organic  composition  of  enemy  units  is  com¬ 
pared  with  the  composition  of  reported  groups  to  classify 
them  into  units.  The  result  of  this  task  are  units. 

2.  Unit  Aggregation 

This  task  deals  with  the  combination  of  units  in  to  higher 
level  units,  sometimes  referred  to  as  higher  level  classifica¬ 
tion.  This  task  uses  known,  organic  models  of  the  composi¬ 
tion  and  behaviour  of  units,  comparative  to  factors  like 
classification,  distance  and  spreading  of  component  units. 
Since  models  are  used  to  perform  this  task,  the  classifica¬ 
tion  of  a  higher  level  unit  is  directly  derived  from  the  classi¬ 
fication  of  the  ‘best  fitting’  model.  The  result  of  this  task  is 
(higher  level)  units. 

3.  Unit  Association 

The  Unit  Association  task  is  concerned  with  associating 
distant  elements  with  known  units.  A  distant  element  can  be 
either  a  single  vehicle  or  a  subordinate  unit.  These  associa¬ 
tions  can  be  used  to  support  other  situation  assessment 
tasks,  e.g.  the  association  of  an  advanced  reconnaissance 
vehicle  with  a  unit  gives  an  indication  of  the  route  to  be 
taken  by  that  unit.  Knowledge  of  organic  composition  and 
disposition  will  be  used  to  determine  the  certainty  of  associ¬ 
ations.  This  task  provides  extended  units. 

4.  State  Estimation 

This  task  deals  with  the  estimation  of  the  current  state  of 
enemy  units,  i.e.  determine  the  disposition  and  activity  of 
enemy  units.  The  location,  velocity,  and  direction  of  a  unit 
is  derived  from  the  known  locations,  velocities,  and  direc¬ 
tions  of  its  elements,  an  element  being  either  a  vehicle  or  a 
unit  itself.  This  task  also  associates  units  with  tracks.  The 
activity  of  unit  is  derived  by  comparing  the  disposition  of 
the  elements  of  a  unit  with  the  organic  disposition  related  to 
organic  activities  of  the  enemy.  The  result  of  this  task  is  an 
estimated  disposition  and  activity. 

5.  State  Prediction 

This  task  deals  with  the  estimation  of  the  future  state  of 
units,  relative  to  known  possible  targets  or  objectives,  i.e. 
determine  the  future  disposition  and  activity  of  enemy  units 
by  reasoning  backward  from  potential  targets.  This  way  the 
probability  of  the  intention  to  capture  the  different  targets 
can  be  estimated  from  the  path  to  be  taken  from  the  current 
position  to  a  target,  and  factors  like  general  direction  of 
enemy  movement,  tracks  and  composition  of  units,  and  cur¬ 
rent  activity. 

6.  Based  upon  the  results  in  task  1-5,  the  Situation  Assessment 
process  determines  a  set  of  questions  for  additional  infor¬ 
mation  requirements,  which  are  sent  to  the  Sensor  Manage¬ 
ment  process.  The  questions  include  e.g.  the  extra  need  for 
information  in  certain  areas,  which  could  lead  to  e.g.  sensor 
mode  changes  or  setting  up  of  drone  plans  by  sensor  man¬ 
agement 

2.4  SENSOR/PLATFORM  MANAGEMENT 

This  process  provides  the  TAD  with  the  capabilities  to  perform 
Sensor/Platform  Management  on  basis  of  the  battlefield  situa¬ 
tion  and  desires  of  the  operator.  A  sensor  management  plan  is 
maintained  and  continuously  updated  which  is  the  basis  for 
platform  and  sensor  management  and  control.  The  following 
types  of  sensor  plus  their  platforms  are  managed:  (1)  Helibome 
sensors,  (2)  Ground  based  sensors,  and  (3)  Drone  sensors. 

The  Sensor/Platform  Management  process  provides  the  fol¬ 


lowing  capabilities: 

1 .  Default  sensor  management  plan 

The  default  sensor  management  plan  is  an  initial  plan  for 
sensor  management  and  reflects  a  long-term  sensor  man¬ 
agement  strategy,  whereas  the  others  below  affect  sensor 
management  on  short  term.  The  default  plan  is  off-line 
defined. 

2.  Prioritized  requests  for  sensor  information  or  direction 
The  requests  may  be  received  from  the  Situation  Assess¬ 
ment  process  and  can  be  modified  by  the  operator.  They  are 
prioritized  indicating  how  demanding  the  wanted  informa¬ 
tion  is.  A  request  is  a  query  expressing  a  conjunction  of  the 
following  elements: 

a.  Battlefield  area; 

b.  Platform  (which  platform  should  be  used  for  observa¬ 
tion); 

c.  Sensors  (which  types  of  sensor  should  be  used  for 
observation); 

d.  Sensor  mode  (which  sensor  mode(s)  should  be  used; 

e.  Time  interval  (when  should  observation  take  place). 

3.  ‘Hard’  directives  for  sensor  control  from  operator 

This  data  is  treated  as  of  highest  priority  and  is  a  mean  to 
override  or  by-pass  the  sensor  management  plan.  The  sen¬ 
sor  management  function  adapts  the  sensor  management 
plan  accordingly. 

4.  Sensor  performance  data 

The  sensor  performance  data  is  received  from  the  (simu¬ 
lated)  sensors/platforms.  To  maintain  an  optimal  sensor 
management  plan  (i.e.  allocation  of  sensors  and  selection  of 
operational  mode)  and  therefore  to  ensure  maximum  acqui¬ 
sition  of  relevant  battlefield  information,  sensor  perform¬ 
ance  is  taken  into  account. 

5.  Meteorological  and  geographical  data 

This  data  enables  the  sensor  management  system  to  con¬ 
strain/adapt  the  plan  to  specific  weather  conditions  (e.g.  fog 
degrades  IR)  and  terrain  conditions  (e.g.  LOS  constraints). 

6.  Interactive  drone  usage  simulation/mission  planning 

This  capability  provides  the  operator  the  support  to  plan  for 
a  drone  mission  during  the  TAD  simulation.  TAD  will  pro¬ 
vide  the  operator  with  a  provisional  flight  pattern  /  plan  for 
the  drone  based  on  the  results  from  the  Data  Fusion  /  Situa¬ 
tion  Assessment.  The  pattern  will  also  be  influenced  by 
enemy  threats.  The  enemy  threats  are  limited  to  a  small 
number  and  are  considered  to  be  static,  i.e.  no  moving 
enemy  threats  (to  meet  calculation  constraints).  The  opera¬ 
tor  is  able  to  update  or  modify  the  drone  flight  plan.  The 
plan  contains:  platform  and  sensors,  list  of  trajectory  points, 
speed  on  every  leg,  and  start  time 

After  specifying  the  mission  plan,  this  capability  checks  the 
mission  plan  on  the  following  constraints: 

a.  Availability  of  the  drone. 

b.  The  observation  points  are  within  the  battlefield  area. 

c.  Maximum  speed 

d.  Maximum  mission  time:  30  minutes 

If  the  drone  mission  plan  is  correct,  it  is  input  for  and  used 
by  the  drone  mission  control  capability.  This  capability 
shall  also  provide  means  to  store  and  retrieve  drone  mission 
plans  under  a  unique  name. 

7.  Drone  mission  control 

This  capability  controls  the  drone  mission  operations  on 
basis  of  the  drone  mission  plan  provided  by  the  drone  mis¬ 
sion  planning  capability.  Drone  control  directives  are  issued 
(in  near  real-time)  to  the  simulated  drone  platforms. 
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8.  Sensor  control 

This  capability  shall  determine  the  time,  place  and  mode  of 
sensor/platform  operations  (excluding  drones)  based  on  the 
data  as  captured  by  the  data  processing  capability.  This  will 
be  done  by  means  of  construction  and  maintenance  of  a  sen¬ 
sor  management  plan.  Based  on  this  plan,  the  sensor  control 
capability  shall  distribute  sensor/platform  control  directives 
to  the  (simulated)  sensors  and  platforms.  The  sensor  control 
directives  support  specification  of  changes  at  some  time  t  to 
the  following  sensor/platform  attributes:  position,  orienta¬ 
tion  and  operational  mode. 

9.  Sensor  management  plan  presentation 

The  system  provides  the  operator  with  the  capability  to 
retrieve  and  display  sensor  management  plan  characteris¬ 
tics.  It  displays  historical  movements,  aiming  and  ranges 
(i.e.  sensor  coverage)  of  sensor  platforms,  their  current 
position  (as  known  to  the  sensor  management  system)  and 
operational  mode,  and  their  trajectories  and  aiming  planned 
in  the  future.  The  system  provides  the  operator  with  the 
capability  to  project  (a  selection  of)  this  information  on  the 
battlefield  area  picture. 

10. Drone  mission  plan  presentation 

The  system  provides  the  operator  with  the  capability  to 
retrieve  and  display  drone  mission  plan  characteristics.  It 
graphically  displays  the  planned  trajectory  and  the  sensor 
coverage  (as  a  function  of  time)  and  current  position  (as 
known  by  the  sensor  management  system).  The  system  pro¬ 
vides  the  operator  with  the  capability  to  project  this  graphi¬ 
cal  information  on  the  battlefield  area  picture. 


2.5  HARDWARE  AND  SOFTWARE  ENVIRONMENT 


Item 

Used  environments 

Workstation 

Sun  SPARCstation 

Operating  system 

Solaris  2.5 

Architecture 

Splice  ’96 

Standard  Languages 

C.C++ 

Functional  Language 

Haskell 

KBS  tool 

(Fuzzy)Clips 

Graphic  tool 

DtBuilder,  Xforms 

GIS 

Data  base  from  MOD  France 

Word  processor 

FrameMaker  on  Sun 

Documentation 

DOD-2167A  (only  significant  parts) 

Knowledge  engineering 

KADS 

CASE 

TeamWork,  method:  Yourdon 

OMT 

Configuration  management 

Standard  Unix  tools 

Project  management 

MS-project 

Quality 

Procedures  defined  for  TA10 

3.  PRESENT  STATUS 

The  TA10  study  is  an  on-going  study  project  which  will  not  be 
finished  before  summer  1998.  In  this  paragraph  the  results  and 
conclusions  for  the  present  status  of  the  project  is  given.  Today 
the  following  components  are  designed  and  implemented: 

1.  Architecture 

2.  World  Model:  databases  and  knowledge  bases 

3.  Battlefield  simulator  &  intervisibility  calculations 

4.  Sensor  simulator 

5.  Fusion  nodes: 


GDFSp  mainly  based  on  extended  Kalman  filtering 
and  MHT  techniques 

GDFS3:  based  on  heuristics  (rule  based  approach) 

-  HDFS 

6.  Sensor/Platform  Management 

From  the  list  above  the  Architeture,  Sensor  Simulation,  and 
Sensor/Platform  Management  are  already  integrated.  In  the 
next  step  (before  the  end  of  1997)  the  others  will  be  added  as 
well  as  the  yet  still  unfinished  modules.  In  the  remaining 
period  the  TA10  study  will  concentrate  on  tests  and  evaluation 
of  the  system.  Furthermore,  the  gap  between  the  near  real-time 
character  of  the  TAD  and  a  real-time  system  will  be  investi¬ 
gated. 
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LIST  OF  ABBREVIATIONS 

BATS 

BATtlefield  Simulator 

CSCI 

Computer  Software  Configuration  Item 

DFS 

Data  Fusion  System 

DFSA 

Data  Fusion  /  Situation  Assessment 

DIR 

Drone  equipped  with  IR  camera 

DP 

Drone  Platform 

DTV 

Drone  equipped  with  TV  camera 

ESM 

Electronic  Support  Measures 

FEBA 

Forward  Edge  of  the  Battle  Area 

FLOT 

Forward  Lines  of  Own  Troops 

GBPX 

Ground  Based  Platformx 

GDF 

Global  Data  Fusion 

GDFS 

Global  Data  Fusion  System 

GDFSX 

Ground  based  Data  Fusion  Systemx 

GGBDF 

Global  Ground  Based  Data  Fusion 

GGBHDF 

Global  Ground  Based  and  Heliborne  Data  Fusion 

GIR 

Ground  based  IR  camera 

GMTIX 

Ground  based  medium  range  MTI  radarx 

GSMGT 

Global  Sensors  ManaGemenT  system 

GTV 

Ground  based  TV  camera 

GTV+IR 

Ground  based  IR  +  TV  camera 
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HDFS 

Helibome  Data  Fusion  System 

HESM 

Heliborne  Electronic  Support  Measures 

IIMTI 

Heliborne  long  range  MT  radar 

HP 

Heliborne  Platform 

IEPG 

Independent  European  Program  Group 

IR 

Infra  Red 

IS 

Input  Simulator 

LGSMGT, 

,  Local  Ground  based  Sensors  Management  system. 

LHSMGT 

Local  Heliborne  Sensors  Management  system 

LOS 

Line-Of-Sight 

MTI 

Moving  Target  Indicator 

NIS 

Nato  Identification  System 

OPFOR 

OPposing  FORces 

SA 

Situation  Assessment 

SM 

Sensor  Management 

STANAG 

STandard  Nato  AGreement 

TAD 

TA10  demonstrator  study 

UTM 

Universal  Transverse  Mercator 
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AIRCRAFT  SENSOR  DATA  FUSION:  AN  IMPROVED  PROCESS 
AND  THE  IMPACT  OF  ESM  ENHANCEMENTS 

C.A.  Noonan  and  M.  Pywell 
British  Aerospace  Military  Aircraft  and  Aerostractures, 

W392D,  Warton  Aerodrome,  Warton,  Lancashire,  PR4  1AX,  U.K. 


SUMMARY 

British  Aerospace  Military  Aircraft  and  Aerostractures  (BAe 
MA&A)  has  conducted  various  studies  on  the  topic  of  Data 
Fusion.  This  paper  highlights  developments  which,  it  is 
thought,  offer  a  significant  step  towards  optimum  situation 
awareness.  It  examines  sensor  data  quality,  the  fusion  process 
and  required  improvements,  and  proposes  a  method  for 
improving  the  quality  of  the  resultant  threat  identification 
function.  The  paper  also  examines  the  key  issues  affecting  the 
quality  of  track  data  fed  into  the  fusion  process  by  Electronic 
Support  Measures  systems  (ESM),  the  prime  contributor  of 
threat  identity  data.  Emitter  recognition  and  location  issues  are 
discussed  and  potential  routes  are  proposed  to  attain  the 
necessary  performance  increases  to  support  optimum  situation 
awareness. 

1  INTRODUCTION 

Swift  and  unambiguous  identification  of  hostile  weapon  system 
type  and  location  in  tactical  scenarios  is  fundamental  to  aircraft 
mission  success  and  survivability.  To  attain  the  highest 
probability  of  mission  success  the  same  identification  and 
classification  quality  is  required  for  all  players  in  the  scenario, 
whether  hostile,  friendly  or  neutral. 

The  fusing  of  data  from  multiple  on-board  multi-spectral 
sensors  and  other  off-board  data  sources  offers  great  potential 
for  achieving  high  grade  situation  awareness.  Example 
on-board  sensors  include  radar,  IFF,  ESM,  Forward-Looking 
and  Search/Track  Infra-Red  (FLIR/IRST),  LIDAR  and  Missile 
Warning  Systems.  This  fusing  corresponds  to  optimisation  of 
own  weapon  targeting,  threat  evasion  and  countermeasures 
capabilities.  These  aspects  can  lead  to  improvements  in 
aircraft  lethality  and  survivability  which,  in  turn,  influence 
affordability,  flexibility  and  availability.  These  five  factors  will 
be,  arguably,  the  key  product  differentiators  in  the  military 
aircraft  market  place  of  the  future. 

This  paper  defines  Identification'  in  the  military  sense  and 
discusses  current  identification  processes  and  limitations.  It 
describes  the  Identity  Fusion  Process  and  the  contribution  of 
various  levels  of  ESM  capability  to  that  process.  Potential 
improvements  to  identification  are  then  described,  comprising 
an  improved  fusion  process  and  those  relating  to  enhanced 
ESM  performance.  Identity  Fusion  Process  simulation  results 
are  presented  for  different  ESM  capability  levels,  conclusions 
are  drawn  and  a  way  ahead  proposed.  Although  the  issues 
addressed  are  applicable  to  all  classes  of  military  aircraft,  the 
focus  of  work  to  date  has  been  on  fighter-sized  aircraft  of  the 
present  and  future. 

2  IDENTIFICATION 

Reliable  identification  of  own  forces,  threats,  non-combatants 
and  targets  poses  problems  on  any  battlefield.  This  is 
particularly  so  in  the  air  where  participants  in  the  battle  may  be 
highly  dynamic  and  where  own  forces  and  hostile  forces  may  be 
interspersed. 

In  order  to  avoid  increasing  the  risk  of  being  the  perpetrator  or 
the  victim  of  fratricide,  aircraft  must  improve  their  abilities  to 
declare  their  identity  to  friendly  forces  and  to  recognise  the 
declarations  of  others.  To  avoid  collateral  damage  and 


casualties  among  non-combatants  the  ability  to  identify  an 
intended  target  or  a  major  threat  positively  before  weapon 
release  must  be  improved.  The  potential  resources  available  to 
an  identification  system  and  the  contributions  they  make  to 
tactical  situation  awareness  are  summarised  in  Table  1 . 

When  own  forces  are  well  separated  from  enemy  forces  and 
non-combatants,  they  may  be  identified  by  their  actions 
conforming  to  some  known  mission  plan.  If  a  package 
encounters  a  friendly  unit  where  the  mission  plan  says  it  will 
be  and  at  the  right  time,  the  task  of  the  identification  system 
will  be  relatively  easy.  By  a  similar  argument,  if  forces  are 
engaged  in  activities  which  do  not  correspond  to  the  mission 
plan  or  any  filed  flight  plan,  they  must  be  regarded  as  suspect. 
However,  lack  of  adherence  to  a  known  plan  is  not  a  conclusive 
indicator  of  hostility. 

Own  forces  may  be  identified  by  the  ability  to  sign  on  to  and 
exchange  information  with  data  communications  networks  and 
their  ability  to  make  the  right  responses  to  Co-operative  Target 
Identification  (CH)  systems.  Again,  the  inability  to  make  the 
appropriate  communications  and  responses  adds  weight  to  the 
supposition  of  hostility. 

For  all  participants  and  non-combatants,  sensor  data  from  Non 
Co-operative  Identification  (NCI)  systems  or  from  imaging 
systems  may  contribute  to  an  eventual  successful  identification 
and,  if  they  are  making  Radio/Radar  Frequency  (RF) 
emissions,  these  may  be  identifiable  by  the  ESM.  These  latter 
approaches  attempt  to  recognise  the  unit  in  question  by  its 
appearance  or  by  the  characteristics  of  the  RF  equipment  it 
carries.  However,  the  changing  European  political  situation  and 
the  global  armaments  market  increase  the  likelihood  that 
similar  units  will  fight  on  both  sides  of  any  conflict  and  so 
decrease  the  confidence  which  can  be  placed  in  these 
approaches. 

When  own  forces  are  interspersed  with  enemy  forces,  the  task 
of  reliable  identification  becomes  more  difficult.  Units  which 
are  identified  by  their  participation  in  data  communications 
networks  can  still  be  identified  reliably  if  they  are  capable  of 
declaring  their  position  with  high  accuracy  (e.g.  by  using  the 
Global  Positioning  System),  provided  the  observer  knows  its 
own  position  equally  accurately.  Interspersing  of  forces  may 
pose  difficulties  for  CTI  and  NCI  systems  if  the  sensor 
resolution  of  the  system  is  poor  and/or  own  and  enemy  forces 
lie  in  close  proximity. 

It  is  reasonable  to  assume  that  the  ESM  will  be  optimised  to 
identify  major  threats  from  the  RF  emissions  that  they  make. 
Traditionally,  the  ESM  has  produced  coarse  direction  and 
estimates  together  with  increasingly  reliable  identity  statements 
and  this  combination  of  data  qualities  poses  particular 
problems  for  the  Sensor  Fusion  (SF)  process.  The  identity 
statements  are  of  increasingly  high  value  but,  in  crowded 
scenarios,  cannot  currently  be  unambiguously  associated  with 
an  individual  track.  These  problems  and  potential  solutions  to 
them  provide  the  subject  matter  for  this  paper. 
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3  CURRENT  PROCESSES  AND  LIMITATIONS 

This  section  will  look  at  the  identification  process  from  two 
points  of  view.  Firstly,  it  considers  how  a  conventional  SF 
system  brings  together  the  identity  information  available  to  the 
aircraft.  Secondly  it  considers  current  ESM  and  the  ways  in 
which  they  derive  platform  identity  and  location. 

3.1  The  Sensor  Fusion  Process 

The  SF  process  exists  to  gather  together  all  the  situation  data 
arriving  at  the  platform  in  question  and  to  consolidate  it  into  a 
single  tactical  picture.  The  platform  which  is  of  interest  here  is 
a  fighter  aircraft  engaged  in  an  air  defence  mission  and  the 
following  assumes  that  application.  SF  may  be  regarded  as  a 
two  stage  process,  see  Figure  1.  The  first  stage  is  tracking, 
during  which  all  the  sensor  measurements  referring  to  a 
particular  platform  are  brought  together  over  time.  This  gives 
the  most  complete  and  accurate  estimate  of  the  platform's 
position,  motion  and  status  that  the  measurements  allow,  along 
with  some  definition  of  the  uncertainty  in  that  estimate.  A 
tracking  process  may  use  measurements  from  one  or  more 
sensors  and  the  sensors  may  be  distributed  over  multiple 
aircraft  and  depend  on  communications  links. 

If  it  were  possible  to  gather  all  the  sensor  measurements  at  a 
single  tracking  process  in  a  timely  and  reliable  way,  a  single 
track  database  would  be  produced  and  the  SF  process  would  be 
complete  at  this  stage.  In  practice,  the  constraints  on  the 
system  are  too  great  and  this  does  not  happen.  So,  at  the  end  of 
the  tracking  stage  several  tracking  processes  will  have 
produced  their  own  database,  each  with  its  own  view  of  the 
world,  which  then  must  be  combined. 

The  second  fusion  stage,  Track-to-Track  Fusion,  exists  to 
combine  these  track  databases  into  a  single  fused  database.  To 
do  this  it  must  perform  the  following  three  tasks: 

•  It  must  align  the  tracks  to  the  same  spatial  axis  set  and  time 

There  is  no  guarantee  that  each  tracker  will  use  the  same  axis 
set.  They  may  measure  very  different  platform  attributes  and 
have  very  different  points  of  view,  especially  when  the  sensors 
they  serve  are  carried  on  separate  aircraft.  So  the 
Track-to-Track  Fusion  process  must  perform  any 
transformations  necessary  to  align  the  platform  data  and 
corresponding  uncertainty  definitions  to  a  common  axis  set. 
Also,  each  track  will  have  received  its  last  update  at  a  different 
instant  in  time.  So  track  data  must  be  extrapolated  to  account 
for  motion  since  it  was  last  updated  and  each  uncertainty 
definition  must  be  modified  to  account  for  possible  manoeuvres 
or  changes  in  status  since  the  update. 

•  It  must  deduce  the  number  of  targets  giving  rise  to  the  data 
and  the  nlatform  from  which  each  track  arose  . 

When  data  first  arrives  at  the  Track-to-Track  Fusion  process 
the  number  of  targets  is  unknown.  If  two  sensors  each  reported 
two  targets,  and  if  the  reporting  of  false  targets  is  sufficiently 
unlikely  for  us  to  ignore  it,  we  must  still  consider  the 
possibilities  that  there  are  two,  three  or  four  targets  being 
detected,  jointly,  by  the  sensors.  To  do  this  we  calculate  the 
likelihood  of  each  possibility  and  choose  the  most  likely  one. 
This  process  is  commonly  referred  to  association  or  correlation. 
Usually,  this  stage  of  the  process  will  attempt  to  optimise  some 
figure  of  merit  for  the  association  process  which  is  linked  to  the 
likelihood  of  making  the  right  choice.  Firstly,  an  algorithm 
would  mark  each  pair  of  tracks  arising  from  different  trackers 
as  'feasible'  or  'not  feasible'  based  on  some  statistical  hypothesis 
test.  Secondly,  it  would  calculate  the  figure  of  merit  for  each 
'feasible'  pair.  Finally,  it  would  perform  a  search  through  the 


possible  combinations  of  'feasible'  pairs  and  choose  the  one 
giving  the  best  overall  figure  of  merit.  Some  implementations 
shorten  this  process  by  performing  these  stages  for  new  tracks 
only.  For  established  tracks  in  such  implementations,  existing 
solutions  would  be  retained  until  new  data  were  received  in 
direct  contradiction  to  them. 

•  It  must  form  joint  tracks  and  joint  identity  statements  for 
targets  reported  by  more  than  one  source. 

The  formation  of  joint  tracks  can  range  from  a  simple  approach, 
which  selects  the  best  single  track  to  represent  the  associated 
class,  up  to  more  complex  approaches  which  calculate  an 
optimal  joint  track  using  an  algorithm  based  in  estimation 
theory  (e.g.  minimum  mean  square  error).  Similarly 
approaches  to  joint  identity  estimate  formation  can  range  from 
simple  voting  to  algorithms  based  in  statistical  theory  (e.g. 
Dempster's  orthogonal  sum). 

A  conventional  approach  to  the  implementation  of  Track-to- 
Track  Fusion  would  adopt  a  process  breakdown  similar  to  the 
functional  one  described  above  and  represented  in  Figure  1. 
However,  this  approach  has  limitations[19]: 

•  The  ability  to  produce  an  unambiguous  solution  to  the 
association  question  depends  on  the  scenario.  When  the 
targets  are  dispersed  and  tracks  from  different  targets  are 
unambiguously  separate,  no  problems  arise.  Similarly,  when 
similar  targets  are  grouped  very  tightly  so  that  tracks  from  the 
same  class  may  be  interchanged  without  affecting  the  solution, 
no  problems  arise.  However,  when  dissimilar  targets  lie  close 
together  incorrect  associations  may  be  made  which  affect  the 
quality  of  the  fused  picture.  This  is  discussed  by  Blackman 
[1].  In  terms  of  the  likelihoods  described  above,  there  would 
be  conflicting  feasible  solutions  with  similar  likelihood 
scores. 

•  The  estimation  process  may  be  based  on  the  assumption  that 
data  has  been  correctly  associated  and  may  attempt  optimal 
combination  of  the  data  on  that  basis  [1,2,3],  Processes  of  this 
kind,  applied  to  incorrectly  associated  data,  may  produce 
meaningless  results.  Paradoxically,  it  is  the  'optimal' 
algorithms  which  are  the  least  robust  in  this  respect  because 
they  rely  most  heavily  on  the  assumption  of  correct 
track-to-track  associations. 

•  When  incorrect  associations  arise,  they  may  change  over  time 
and  with  them  the  output  of  the  estimation  processes.  This  in 
turn  leads  to  incorrect  and  changing  information  displayed  in 
the  cockpit  with  errors  that  move  from  platform  to  platform 
over  time. 

Data  from  the  ESM  is  particularly  prone  to  problems  of  this 
sort  because,  in  present  day  systems,  it  tends  to  produce  tracks 
with  the  coarsest  positional  accuracy.  At  the  same  time,  the 
identity  statements  it  produces  may  be  the  most  specific  and 
accurate  available  within  the  avionics  system.  Thus,  there  is  a 
great  incentive  to  use  them.  The  result,  when  unfavourable 
scenarios  are  encountered,  can  be  incorrect  and  changing 
identity  statements  displayed  in  the  cockpit.  There  are  several 
ways  in  which  these  limitations  might  be  overcome.  Two  are 
considered  in  this  paper: 

•  improvements  in  sensor  accuracy  and  resolution,  which  would 
restrict  the  problem  to  more  distant,  tightly-grouped 
formations  of  targets. 

•  the  use  of  algorithms  which  recognise  the  potential  for 
ambiguity  and  take  it  into  account,  which  would  prevent  the 
generation  of  incorrect  and  unstable  solutions. 
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3.2  ESM,  its  Contribution  and  Limitations 

Most  threat  weapon  systems  have  a  RF  targeting  and/or 
guidance  component,  usually  in  C-K  band  (0.5-40  GHz)  and 
predominantly  in  E-J  band  (2-18  GHz),  although  there  is  now 
an  increasing  number  of  laser-only  or  laser-augmented  systems. 
For  the  systems  where  RF  is  employed,  the  primary  on-aircraft 
measurement  and  warning  sensor  is  the  ESM.  In  this  paper  the 
term  ESM  is  taken  to  mean  any  level  of  Electronic  Warfare 
(EW)  antenna/receiver  system  capability,  from  simple  Radar 
Warning  Receiver  (RWR),  through  conventional  ESM,  to  the 
most  complex  (and  costly)  Electronic  Intelligence  (ELINT) 
equipment.  A  factor  in  common  between  RWR,  ESM  and 
ELINT  is  their  function  of  detecting  and  .processing  radar 
signals.  The  main  differences  are: 

•  RWRs  are  used  primarily  for  threat  warning. 

•  ESM  is  used  for  threat  warning,  the  detection  and 
identification  of  non-threat  emitters,  such  as  surveillance 
radars,  and  to  determine  emitter  location.  An  ESM  system 
designed  primarily  for  emitter  location  is  called  an  Emitter 
Location  (EL)  or  Locator  System,  such  as  that  fitted  to  the 
Electronic  Combat  and  Reconnaissance  (ECR)  Tornado. 

•  ELINT  is  used  for  the  detection,  recording  and  analysis  of 
radar/radio  signals  as  well  as  locating  emitters.  It  may  be 
implemented  by  the  addition  of  a  recording  and  analysis 
capability  to  ESM,  but  often  uses  more  sophisticated  receiver 
systems.  The  emitter  data  resulting  from  ELINT  analysis  can 
be  entered  into  the  data  bases  which  are  needed  for 
reprogrammable  EW  systems. 

The  data  that  ESM  can  contribute  to  the  fusion  process  are: 

•  Azimuth  Direction  of  Arrival  (DOA)  and,  in  some  instances 
Elevation  DOA, 

•  Emitter,  mode  and  associated  platform  identification,  each 
with  a  recognition  confidence  factor, 

•  Time  of  Arrival  (TOA)  and  Range  to  the  emitter,  and 

•  Priority,  if  the  emitter  is  a  threat 

The  ESM,  dependent  upon  its  capability,  may  also  provide 
measured  emitter  RF  parameters  (e.g.  the  ELINT  fundamental 
parameters  of  frequency.  Pulse  Width  and  Repetition  Interval, 
received  power,  and  scan  time/rate)  and  EL  (from  DOA  and 
range).  Derived  parameters  include  RF  type  (fixed,  hopper, 
deviations,  etc.),  PRI  type  (fixed,  jitter,  stagger,  positions, 
elements,  etc.)  and  Scan  type  (circular,  conical,  etc.).  The 
accuracy  of  DOA,  quality  of  and  confidence  in  the  above 
identification,  and  speed  with  which  the  ESM  determines 
them,  determine  their  importance  to  the  fusion  process. 
Probability  of  signal  Intercept  (POI),  whilst  crucial  to  the 
emitter  recognition  process,  is  not  per  se  an  input  to  the  fusion 
process.  POI  is  usually  specified  for  an  ESM;  with  typical 
values  of  approaching  100%  for  modem  ESM.  These  factors 
are  also  arguably  the  key  performance  and  cost  differentiators 
between  the  sub-classes  of  ESM. 

DOA  Determination 

ESM  systems  use  DOA  determination  techniques  which  are 
based  upon  the  measurement  of  some  combination  of 
amplitude,  phase  and  time  of  arrival  of  an  emitter's  RF  signal  at 
a  number  of  co-located  and/or  remotely  located  receive 
antennas  on  the  airframe.  Current  techniques  are  listed  in 
Table  2  and  are  adequately  described  in  a  number  of  texts,  e.g. 
[4]-[6].  The  resulting  DOA  accuracy  is  primarily  a  function  of 
antenna  type  and  locations,  combined  with  receiver 
measurement  accuracy  of  frequency,  time  and  phase.  Table  2 


summarises  typical  current  DOA  accuracies  for  the  main 
techniques.  It  should  be  noted  that  the  absolute  accuracy  is 
often  different  for  frequency  sub-bands,  dependent  upon  the 
technique  and  receiver  combinations)  used.  Newer  DOA  and 
EL  techniques,  e.g.  differential  Doppler,  are  discussed  later. 

Emitter/Platform  Recognition 

Significant  commonality  of  RF  parametrics  of  hostile  and 
friendly  radars  limits  the  ability  of  current  EW  systems  to 
provide  the  aircrew  with  unambiguous  identification  of  the 
illuminating  emitters.  This  is  exacerbated  by  errors  in  RF 
parametric  measurements  and  shortcomings  of  intelligence  data 
programmed  into  the  ESM.  These  issues,  which  are  expanded 
upon  in  [7],  have  an  adverse  impact  on  situation  awareness  and 
the  timely  deployment  of  electronic  countermeasures.  There 
are  four  inter-related  issues  which  need  to  be  addressed  if 
improved  situation  awareness  is  to  be  achieved  and  increased 
platform  survivability  ensured.  These  arc  the  de-interleaving 
of  incident  RF  pulse  trains,  the  resolution  of  the  fundamental 
problem  of  emitter  ambiguity,  the  precise  identification  of 
platforms,  and  the  potential  benefits  of  the  preceding  items  on 
countermeasures  effectiveness.  Of  these  four,  emitter 
ambiguity  is  seen  as  the  main  issue  and  its  resolution  may  lie 
in  improved  measurement  and  processing  of  intra-pulse 
modulation  on  signals. 

Dense  (>1  MPPS)  RF  environments  can  be  achieved  nowadays 
in  certain  frequency  sub-bands,  especially  now  high-PRF 
pulse-doppler  radars  are  common.  Typical  current  ESM  can, 
within  the  very  short  time  allowed  to  cater  for  countermeasure 
engagement/dispensing,  only  offer  track  file  outputs  as  a  list  of 
potential  solutions  to  which  emitters  it  thinks  it  has  seen, 
prioritised  according  to  some  pre-set  rules  e.g.  hostile  emitters 
are  at  the  top  of  the  list.  Where  association  of  these  emitters  to 
a  platform  can  be  made  by  the  ESM,  that  too  may  be  declared 
on  the  track  file.  The  confidence  of  the  ESM  in  its 
determination  of  the  probability  of  a  given  emitter  and  platform 
declaration  being  correct  is  also  flagged  per  emitter.  For 
current  systems  this  confidence  factor  is  rarely  unity  for  other 
than  the  simplest  of  RF  scenarios. 

Electronic  'fingerprints'  of  emitters  and  emitter  types  are 
discussed  in  a  number  of  texts  e.g.  [8]-[10],  Although  there 
are  differing  interpretations  of  the  term  'fingerprinting',  it 
means  the  use  of  a  unique  set  of  measurable  parameters  which 
enables  either  differentiation  between  emitter  types  (e.g.  by 
features  peculiar  to  the  radar  transmitter  type),  or  between 
emitters  of  the  same  type,  or  indeed  (and  ideally)  both.  This 
topic  is  discussed  later.  A  high  level  of  emitter  'fingerprinting' 
can  be  achieved  using  current  RWR/ESM  (as  opposed  to 
ELINT  systems),  but  only  where  the  RF  environment  is 
relatively  limited.  [10]  describes  such  an  eight  emitter,  I-band 
scenario  where,  with  one  exception,  the  emitters  could  be 
unambiguously  identified  using  today's  ESM  technology 
capabilities  of:  RF  resolution  (5  MHz),  PW  resolution  (50  ns), 
TOA  resolution  (50  ns)  and  Scan  resolution  (4  ms). 

Another  problem  of  ESM  capability  limiting  its  usefulness  to 
the  determination  of  a  real-time  tactical  picture  is  the  update 
rate  of  track  file  information.  The  ideal  ESM  performance 
requirement  is  to  see  only  the  leading  edge  of  the  first  RF 
signal  (pulse  or  CW),  and  to  instantly  and  unambiguously 
recognise  the  emitter,  classify  friend  or  foe  and  instigate  crew 
notification,  chaff  dispensing  and/or  ECM  engagement.  In 
reality  a  few  pulses  and  a  few  seconds  are  required  to  achieve 
the  above  with  any  degree  of  confidence.  In  order  to  provide 
sensible  polar-type  spokes  on  a  CRT  display  the  identified 
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emitter,  its  bearing  and  signal  strength  need  to  be  displayed  for 
a  finite  period  of  time.  To  cater  for  scanning  or  slow  rotation 
rate  emitters,  where  there  may  typically  be  up  to  10  seconds 
between  RF  'wipes'  across  the  ESM  antennas,  the  displays  and 
appropriate  countermeasures  engaged  may  be  kept  on  for  the 
duration  -  until  the  emitter  is  definitely  no  longer  illuminating 
the  host  aircraft.  This  can  lead  to  the  contents  of  the  track  file 
indicating  emitter  presence  for  some  number  of  seconds  after  it 
has  actually  ceased  to  pose  any  form  of  threat,  a  limitation  for 
fighter  operations. 

Emitter  Location 

The  objectives  and  required  accuracies  of  EL  are  summarised 
in  Table  3.  A  number  of  conventional  techniques  exist  for  EL 
[5],  [4],  although  only  three  are  strictly  applicable  to  military 
aircraft  in  the  wingspan/length  range  15  m  (fighters)  to  35  m 
(maritime  patrol  aircraft).  These  techniques  are: 

•  Azimuth/Elevation:  Ground  emitters  can  be  located 
instantaneously,  assuming  the  use  of  aircraft  altitude  and 
reasonable  DOA  accuracy,  either  amplitude  comparison  (low 
cost)  or  interferometer  (high  cost)  direction  finding 
techniques.  The  down-  and  cross-range  errors  are  a  function 
of  DOA  errors,  height  accuracy  and  range  to  emitter.  Table  4, 
adapted  from  [5]  gives  an  indication  of  the  down-range  errors 
for  two  altitudes,  using  an  early  generation  3  azimuth,  3 
elevation  spiral  antenna  interferometer  array.  The  ranging 
accuracy  of  this  technique  is  best  at  high  altitude  (large 
depression  angles)  and  degrades  rapidly  at  low  altitudes  -  an 
appreciable  limitation  for  low  level  operations. 

•  Triangulation:  The  aircraft  takes  azimuth  (or  rather  bearing) 
measurements  at  regular  intervals  of  a  few  seconds,  and  uses 
triangulation  and  estimation  algorithms  to  arrive  at  an 
accurate  EL  within  a  few  seconds.  This  technique  is  more 
applicable  to  the  ECR/ELINT  role  than  to  the  fighter/bomber 
application,  as  it  is  not  a  forward-looking  technique.  It  is, 
however,  used  on  a  number  of  current  systems,  e.g.  that  on  the 
ECR  Tornado,  and  requires  very  accurate  DOA  to  achieve 
useful  ranging  against  airborne  emitters.  Fig.  9  of  [5]  gives  an 
indication  of  range  uncertainty  using  this  technique  and  two 
examples  from  that  figure  indicate  the  limitations  of  this 
technique  for  fighter  applications  as  follows.  To  achieve  a 
reasonable  range  uncertainty  (say  3%)  with  an  EL  system  with 
1°  DOA  accuracy,  at  an  own-aircraft  speed  of  400  Kts  and  at  a 
nominal  emitter  to  aircraft  range  of  15  n.  miles,  then  one 
measurement  would  be  required  every  2  sec.  for  60  sec.  To 
reduce  the  required  measurement  time  to  that  relevant  to 
fighter  operations  (~5  sec.),  a  DOA  accuracy  of  0. 1°  would  be 
required  to  achieve  even  20%  range  uncertainty. 

•  Time  Difference  of  Arrival  (TDOA1:  Traditionally  this  has 
been  a  multi-platform  technique,  but  can  be  implemented 
successfully  on  a  large  aircraft,  now  that  TOA  measurement 
systems  with  1-5  ns  resolution  are  available.  It  is  a  complex 
technique  and  technology,  yielding  high  accuracy  with  high 
speed  -  but  at  high  cost.  It  is  unlikely  to  be  feasible  on  a 
fighter-sized  airframe  due  to  the  need  for  very  wide  spacing  of 
antenna  to  form  TDOA  measurement  baselines. 

In  each  of  the  above  techniques,  various  methods  can  be  used  to 
resolve  location  ambiguities  and  reduce  overall  emitter  position 
error.  One  of  the  simplest  methods  is  range  estimation  by  the 
comparison  of  measured  signal  power  against  that  stored  in  the 
ESMs  emitter  database  for  the  Effective  Radiated  Power  of 
that  emitter.  Paradowski  [11]  discusses  EL  techniques  and 
algorithms,  and  includes  a  number  of  these  methods. 


4  POTENTIAL  IMPROVEMENTS  TO  THE 
IDENTIFICATION  PROCESS 

This  section  will  look  at  possible  improvements  to  the 
identification  process.  Firstly,  it  considers  how  the  SF  process 
might  better  extract  identity  information  from  the  ESM  by 
allowing  for  ambiguity  in  its  calculations.  Secondly,  it 
considers  future  ESM  and  the  ways  in  which  the  quality  of 
platform  identity  and  location  statements  might  improve,  so 
improving  the  intrinsic  capabilities  of  the  system  to  resolve 
ambiguity. 

4.1  The  Improved  Sensor  Fusion  Process 

We  have  developed  an  approach  to  Identity  Fusion  which  takes 
account  of  the  ambiguity  in  the  Track-to-Track  association 
solution.  The  approach  calculates  the  true  probability  of  each 
identity  for  each  track.  When  ambiguity  is  present  it  results  in  a 
solution  which  is  offered  with  lower  confidence  than  a 
conventional  approach  but  which  is  stable  and  relatively  free 
from  error.  In  the  absence  of  ambiguity,  it  produces  a  solution 
which  is  indistinguishable,  numerically,  from  that  produced  by 
the  conventional  approach.  We  will  refer  to  the  new  approach 
as  Joint  Probabilistic  Identity  Fusion.  We  propose  two  changes 
to  the  conventional  SF  process  of  section  3: 

•  The  ESM  'tracks'  will  undergo  a  separate  Track-to-Track 
Fusion  process. 

In  particular,  the  alignment  and  association  processes  will  be 
performed  with  respect  to  a  partial  fused  picture  comprising  all 
non-ESM  data.  This  may  be  expressed  in  terms  of  probabilities. 
Previously,  we  calculated  a  figure  of  merit  which  allowed  us  to 
maximise  the  likelihood  of  choosing  the  correct  set  of 
associations.  The  probability  distribution,  P(X.),  that  our  fused 
picture  is  based  on  the  true  set  of  associations  is  written: 

P(X)=n 

i 

where  X  is  the  fused  picture,  Z  represents  the  set  of  single 
source  tracks  and  x,  is  the  zth  fused  track  in  X. 

In  the  improved  process  we  calculate: 

p(x')=np(X;iz,) 

i 

and 

p(X)=n;>(xIlx/,ze) 

i 

where  X'  represents  the  partial  fused  picture,  x'  is  the  Ith 
partial  fused  track  in  X'and  Ze  and  Zk  represent  the  set  of 
ESM  tracks  and  the  set  of  other  tracks  respectively. 

This  change,  in  itself,  does  not  improve  matters  greatly.  The 
true  value  of  P(X)  should  not  change.  However,  it  imposes  a 
processing  structure  into  which  our  Joint  Probabilistic  approach 
fits  neatly.  Thus,  it  enables  us  to  calculate  a  better 
approximation  to  P(X) . 

Also,  it  is  worth  noting  that  situations  leading  to  uncertainty 
about  P(X')  are  relatively  rare  and  we  are  able  to  obtain  a 
good  approximation  to  this  distribution  whilst  avoiding  a  large 
amount  of  redundant  processing.  The  key  factor  in  the  accuracy 
of  the  association  process  is  the  distribution  of  physical  targets 
and  this  change  delays  ESM  association  until  the  most 
complete  statement  of  this  distribution  is  possible. 

•  Association  and  identity  estimation  will  adopt  a  Joint 
Probabilistic  approach  f21. 

This  means  that,  during  association,  instead  of  associating  ESM 
tracks  with  targets  on  a  one-to-one  basis,  we  will  produce 
probabilities  that  each  ESM  track  arose  from  all  of  the  targets 
present  in  the  scenario.  We  can  calculate  this  with  a  high 
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degree  of  certainty  using  standard  statistical  theory.  What  we 
cannot  do  with  any  certainty  is  choose  which  one  platform  the 
ESM  track  arose  from  when  these  calculations  yield  similar 
probabilities  for  several  tracks.  The  advantage  of  this  approach 
is  that  we  avoid  this  choice. 

Previously  we  chose  a  set  of  associations.  © ,  such  that 
©  =arg  max  {P(Xl©/)} 
j 

where  ©/  is  the  f‘  feasible  set  of  track-to-track  associations. 
Then  P(X I©)  was  our  (often  poor)  approximation  to  P(X) . 

In  the  improved  process,  we  continue  to  use  this  approach  for 
non-ESM  tracks  to  obtain  the  set  of  partial  associations  ©7 
noting  that  P(X/I©/)  is  nearly  always  a  good  approximation 
for  PCX') . 

For  ESM  tracks  we  calculate  the  probability: 

6;/=/>(x,  =zel) 
where  zej  is  the  /“’ESM  track. 

During  identity  estimation,  instead  of  combining  associated 
identity  statements  under  the  assumption  of  correct  association, 
we  will  combine  all  ESM  identity  statements  into  each 
platform,  weighted  by  its  probability  of  association. 

Previously  the  probability  of  each  identity  was  calculated  for 
fused  track  i  using  Bayes  rule: 

p(ym  |Z,  0)  =  p(ym)x  n  p(y,„  |z,„)  +p( Z) 

where  ym  is  the  m'h  feasible  identity  and  z,„  is  an  identity 
statement  from  sensor  n  associated  with  fused  track  i  by  ©. 
Then  the  identity  of  track  i  was  chosen 

if  =arg  max  {p(ym  |Z,©)} 

m 

and  a  declaration  was  made  provided  the  associated  probability 
exceeded  some  threshold.  This  worked  well  only  when 
.P(X|@)  proved  to  be  a  good  approximation  to  P(X)  • 

In  the  improved  process  the  probability  of  each  identity  is 
calculated: 

p(ym\Z,@',Q) 

p(ym)xllp(ym\zin)  xE  [p(y  Jze, ,6,, )\+p(Z) 

where  zei  the  /th  ESM  track.  Thus,  the  probability  of  each 
identity  no  longer  depends  on  the  unreliable  ESM  associations 
because  we  have  not  chosen  ESM  associations  on  the  basis  of 
questionable  data.  The  most  likely  identity  will  be  chosen  : 
i,  =argmax  {p(ym IZ,©'^)} 

m 

and  a  declaration  made  or  withheld  in  the  same  way  as  before. 

This  change  calculates  the  true  probability  of  each  identity 
class  for  each  platform.  When  no  ambiguity  is  present,  this  will 
produce  the  same  clear  identity  statement  as  before.  When 
ambiguity  is  present  it  will  be  reflected  in  a  broader  spread  of 
probable  identity  classes  for  each  affected  platform  and  as  a 
result  it  will  be  obvious  that  a  confident  statement  of  identity 
cannot  be  given.  However,  it  may  be  possible  to  make  more 
general  statements  of  identity  in  these  circumstances. 

The  improved  SF  process  may  be  visualised,  see  Figure  2. 
Where  a  conventional  process  would  produce  incorrect  and 
unstable  identity  statements,  we  assert  that  the  Joint 
Probabilistic  Identity  Fusion  process  will  allow  correct  and 
stable  generalised  statements  to  be  made. 


4.2  Improvements  via  Enhanced  ESM  Performance 

The  contribution  of  ESM  to  the  data  fusion  process  is  currently 
undergoing  a  step  improvement.  Publications  by  ESM 
suppliers  suggest  that  the  ideal  ESM  performance  of 
instantaneous  unambiguous  identification  and  exact  spatial 
location  of  pulse/CW  emitters  may  soon  be  feasible  on  any  size 
of  platform.  Moreover,  the  technologies  and  techniques 
currently  under  development  appear  to  eventually  be  applicable 
as  relatively  low  cost  retro-modification  kits'  to  existing 
capability  RWR/ESM.  The  three  key  development  areas  are: 

•  Advanced  Combinational  EL  Techniques:  Ongoing  advances 
in  processing  speeds  and  processing  technologies  now  enable 
combinations  of  classical  DOA  and  EL  techniques,  in  order  to 
optimise  EL  performance,  minimise  errors  and  mitigate  the 
shortfalls  of  individual  techniques.  A  good  example  of  this  is 
the  integrated  ranging  technique  in  the  Litton  Digital  Receiver 
[12],  developed  under  the  U.S.  Precision  Location  And 
IDentification  (PLAID)  programme  [13]-[14],  which  combines 
long  baseline  (phase  rate  of  change),  TDOA,  frequency 
Doppler  and  time  Doppler.  These  combinations  can  also 
include  novel  techniques  such  as  Differential  Doppler  [4], 
[11],  which  have  only  become  realistic  techniques  for  fighter 
aircraft  with  the  advent  of  receivers  capable  of  measuring 
frequencies  to  fractions  of  Liz  [14], 

•  Fingerprinting':  Current  ESM  have  limited  capability  to 
quickly  and  unambiguously  identify  emitters,  especially  when 
the  RF  environment  is  dense.  A  number  of  agencies  and 
research  programmes  have  been  addressing  this  fundamental 
limitation  for  many  years.  Only  recently  have  technological 
developments  occurred  which  now  are  believed  to  offer  hope 
of  achieving  the  above  goal.  During  this  time  the 
proliferation  of  high  performance  and  complex  (multi-mode, 
high  PRF)  radars  has  continued,  with  a  1996  estimate  of  4030 
different  radar  types  world- wide  [15].  Close  to  ideal  ESM 
capability  is  now  believed  to  be  feasible  by  using  digital 
receivers  (see  below)  together  with  combinations  of  analysis 
techniques  such  as  classical  parameter  (frequency,  PW,  PRI 
etc.),  clock  de-interleaving,  fine  grain  Intra-Pulse, 
Unintentional  Modulation  on  Pulse  and  EL  analysis  [13]-[14], 
Supplier  claims  [16]  suggest  that,  by  the  year  2010,  100%  of 
emitter  ambiguity  resolution  may  be  resolvable  as  shown  in 
Table  5,  where  SEI  =  Specific  Emitter  Identification. 

•  Digital  Receivers:  To  enable  the  above,  receiver/measurment 
system  improvements  have  been  required.  A  new  generation 
of  digital  receivers  have  thus  been  developed,  of  which  [12] 
and  the  Lockheed-Martin  Passive  Ranging  Subsystem  (PRSS) 
[17]  are  examples,  approach  readiness  for  in-service  use. 
These  have  very  high  measurement  accuracies/resolutions  for 
frequency  (<10  Hz),  phase  (few  degrees),  TOA  (2  ns  or 
better),  PRI  (sub-ns)  and  amplitude  (<1  dB). 

Other  potential  ESM  enhancements  include: 

•  higher  receiver  sensitivity  (better  than  -60  dBm  [10]), 
improved  signal-to-noise  and  the  use  of  more  efficient  spiral 
antennas,  e.g.  the  spiral  microstrip  type  [18],  These  would 
equate  to  improved  emitter  detection  range  and  could  further 
assist  in  ambiguity  resolution  by  comparison  of  measured 
signal  power  vs.  emitter  database  effective  radiated  power. 

•  use  of  artificial  intelligence  (Knowledge-Based  Systems),  cf 
Ch.8.7  of  [6].  At  the  simplest  level,  ambiguity  resolution 
could  be  aided  by  masking  by  logical  aspects,  e.g.  a)  ship 
radars  don't  fly;  and  b)  if  it's  in  front  AND  is  above  you  AND 
has  a  high  PRF  AND  is  coming  this  way  THEN  it's  very 
highly  likely  to  be  a  threat! 
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5  SIMULATION  RESULTS 

Detailed  simulations  were  performed  and  two  scenarios  were 
examined,  see  Figure  3.  The  first  is  an  unambiguous  Combat 
Air  Patrol  (CAP)  scenario  capable  of  resolution  using  a 
conventional  SF  algorithm  and  current  ESM.  The  second  is  an 
ambiguous  CAP  scenario  which  is  not.  The  scenarios  are 
identical  with  the  exception  of  the  addition  of  a  bomber 
formation  to  the  latter,  which  is  sufficient  to  introduce  the 
ambiguities  discussed  previously.  Each  CAP  aircraft  is 
equipped  with  Radar,  IRST  and  ESM,  and  they  exchange  track 
information  using  data  links.  The  Radars  are  assumed  to 
contribute  identity  information  in  the  form  of  a  size  estimate 
and  a  Jet  Engine  Modulation  measurement.  Data  link  tracks 
contain  the  transmitting  platform's  estimate  of  identity  based  on 
locally  sensed  data.  The  ESM  identity  statements  are  assumed 
to  be  the  most  specific  and  accurate  in  the  system. 

Performance  was  measured  using  the  conventional  Identity 
Fusion  process,  as  described  in  section  3,  and  the  Joint 
Probabilistic  Identity  Fusion  process  and  with  'current'  and 
'2010'  ESM.  Results  are  presented  as  plots  of  %  tracks  a) 
correctly  identified  (i.e.  Tornado,  Hawk  etc.),  b)  placed  in  the 
correct  class  (fighter,  bomber  etc.),  c)  not  identified,  and  d) 
wrongly  identified  or  classified.  Basic  identification  as  hostile, 
friend,  neutral  was  not  simulated  but  the  same  principles  apply. 
The  results  indicate  the  performance  of  the  system  with  respect 
to  identification  of  hostile  aircraft,  hi  all  cases,  friends  reported 
their  identity  and  position  (with  GPS  accuracy)  via  data  links 
and  were  unambiguously  identified. 

Unambiguous  scenario  results: 

■  Figure  3.1  shows  the  performance  of  the  conventional  SF 
process  with  the  'current'  ESM  system.  After  ~30  seconds  of 
the  scenario  every  track  was  correctly  identified  or  classified. 
After  — 100  seconds  it  was  possible  to  identify  every  platform. 

•  Figure  3.2  shows  the  performance  of  the  improved  SF  process 
with  the  'current'  ESM.  There  is  a  slight  improvement  but  the 
result  is  broadly  similar  to  Figure  3.1. 

•  Figure  3.3  shows  the  performance  of  the  conventional  SF 
process  with  the  '2010'  ESM  system.  This  combination  of 
systems  identifies  the  targets  quickly  and  fully. 

Ambiguous  scenario  results:  The  time  windows,  which  are 
different  for  each  scenario,  were  those  during  which  sensor 
coverage  of  the  hostile  aircraft  was  or  approached  its  maximum 
and  thus  most  revealing  of  the  SF  process  performance. 

•  Figure  3.4  shows  the  performance  of  the  conventional  SF 
process  with  the  'current'  ESM  system.  This  combination 
never  fully  identified  the  hostile  targets.  20%  or  more  of  the 
identities  were  in  error  and  the  errors  were  unstable,  moving 
from  aircraft  to  aircraft. 

•  Figure  3.5  shows  the  performance  of  the  improved  SF  process 
with  the  'current'  ESM.  All  aircraft  are  correctly  identified 
after  ~80  sec. 

•  Figure  3.6  shows  the  performance  of  the  conventional  SF 
process  with  the  '2010'  ESM  system.  Here,  all  targets  were 
identified  but  the  time  taken  (—120  seconds)  to  resolve  all 
identities  was  longer  than  in  Fig.  3.5.  The  simulation  used  did 
not  Kalman  filter  the  ESM  tracks  prior  to  Track-to-Track 
association.  Had  it  done  so,  performance  would  have  been 
better  because  the  resulting  sight-line  angular  velocity 
information  would  have  made  it  easier  to  discriminate 
between  the  tracks  and  convergence  to  the  fully  identified 
state  would  have  been  quicker. 


6  CONCLUSIONS  AND  WAY  AHEAD 

The  simulations  showed  that  the  proposed  Joint  Probabilistic 
Identity  Fusion  process  and  the  '2010'  ESM  system  were  both 
capable  of  resolving  the  kinds  of  ambiguity  which  would  defeat 
conventional  systems.  Combined,  they  would  be  capable  of 
dealing  with  greater  and  more  complex  ambiguities.  They  also 
showed  that  the  Joint  Probabilistic  approach  to  Identity  Fusion 
also  promises  improved  performance  in  aircraft  with  an  earlier 
generation  ESM.  Suggested  future  research  paths  are: 

•  examination  and  refinement  of  this  approach  using  real  sensor 
data,  in  a  suitable  rig  and  (subsequently)  aircraft  environment. 

•  further  development  and  refinement  of  the  apporach  and 
algorithms  via  inclusion  of  a  model  of  a  year  '2010'  ESM. 

•  inclusion  of  'Smart'  sensor  systems  in  the  simulation.  In  our 
simulations,  the  opposing  radars  were  dumb  and  noisy.  Smart 
sensor  systems  which  integrate  data  from  multiple  sensors  and 
make  Radar  emissions  only  when  it  is  absolutely  necessary 
are  feasible  using  today's  technologies.  Such  systems  will 
increase  the  uncertainty  surrounding  ESM  data  by  reducing 
the  data  rate  and  will  militate  further  for  the  use  of  an 
approach  to  Identity  Fusion  like  the  Joint  Probabilistic  one 
described  here. 
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Table  1:  Identification  System  Resources 


Resource: 

Own 

Forces 

Threats 

Targets 

Non- 

Combatant 

Mission  Plan 

0 

0 

0 

0 

Communications  with 
own  forces. 

* 

* 

* 

IFF  (CH) 

0 

0 

0 

0 

Long  range  imagery 

0 

0 

0 

0 

RADAR(NCTI) 

0 

0 

0 

0 

LIDAR(NCTI) 

0 

0 

0 

0 

ESM 

* 

* 

* 

0 

Self-defence  systems 

* 

*  ..  may  provide  conclusive  identity  statement, 
o  ..  can  contribute  identity  information. 


Table  2:  Direction  of  Arrival  Technologies  and  Accuracies 

(developed  from  [4]  and  [5]) 


Antenna 

Configuration 

Typical  r.m.s. 
Accuracies 

Comment  (regarding 
application  to  aircraft) 

Amplitude 

Comparison 

3-15° 

Minimum  capability  of  all 
sub-classes  of  ESM. 

Phase 

interferometer 

0.1-3° 

Forward  coverage,  azimuth 
only  (some  elevation  also). 

Spinner 

2-5° 

Not  appropriate  to  fast  jets. 

Multibeam 

1.7-2° 

Naval  and  ground  vehicle 
applications. 

Time  Difference 
Of  Arrival 

<2° 

Complex;  needs  large 
airframe  for  highest 
accuracy. 

Table  3:  Implications  of  Emitter  Location  Objectives 

(from  [4]) 


Objective 

Enables 

Required 

Accuracy 

Electronic  Order 
of  Battle 

Location  of  emitter  types 
associated  with  specific 
weapons  and  units  show 
enemy  strength,  deployment 
and  mission. 

Medium: 

1km 

Weapon  sensor 
location  (Self- 
Protection) 

Focusing  of  jamming  power 
or  manoeuvre  for  threat 
avoidance 

Low:  general 
angle  and 
range  ~5km 

Weapon  sensor 
location  (Protect 
Friends) 

Threat  avoidance  by  other 
friendly  combatants 

Medium: 

—lkm 

Enemy  asset 
location 

Narrowed  recce  search  or 
handoff  to  homing  devices 

Medium: 

~5km 

Precision  target 
location 

Direct  attack  by  'dumb 
bombs'  or  artillery 

High:  ~100m 

Emitter 

differentiation 

Sorting  by  location  for 
separation  of  threats  for 
identification  processing 

Low:  general 
angle  and 
range  ~5km 

Table  4:  Typical  Position/Range  Determination  Accuracies 
using  conventional  single-aircraft  EL  Techniques 

(Adapted  from  [5]) 


Range  Error 

Plan  Range 
n.  miles 

n.  miles 

% 

5 

0.43 

8.6 

0.07 

1.4 

10 

1.75 

4 

20 

7 

30 

KOI 

40 

- 

- 

50 

- 

- 

10.9 

21.8 
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Table  5:  Year  2010  Scenario  Ambiguity  Resolution 
(from  [16]) 


*  SEI. 


Figure  3:  Test  Scenarios 
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1.  SUMMARY 

We  have  studied  remote  sensing  data  from  sensors  in 
different  wavelength  regions  (optical,  thermal  infrared 
and  microwave)  and  from  different  platforms  (airborne 
and  spaceborne)  in  order  to  extract  geographical 
information.  By  comparing  the  extracted  information 
with  an  existing  geographical  database  of  a  test  area  in 
the  Netherlands  we  find  that  to  obtain  military  relevant 
cartographic  information  from  remote  sensing  images 
resolutions  of  5  meter  or  less  are  required.  For 
appropriate  classification  of  extended  objects  like 
agricultural  fields  multi-layer  imagery  is  necessary. 

2.  INTRODUCTION 

Due  to  the  changing  international  situation  after  1989 
the  tasks  of  the  Royal  Netherlands  Army  (RNLA)  have 
changed  considerably.  Nowadays  the  RNLA  considers 
and  carries  out  operations  outside  the  actual  NATO 
area  contrasting  the  situation  during  the  Cold  War.  For 
NATO  areas  geographical  information  is  sufficiently 
available  and  in  case  of  allied  operations  this 
information  is  shared.  For  many  other  parts  of  the 
world  geographical  information  is  often  sparse,  and  for 
non-NATO  operations  like  UN  operations  each  country 
is  responsible  for  its  own  intelligence.  In  this  context  an 
independent  and  accessible  source  of  geographical 
information  is  a  necessity.  Remote  sensing  data  from 
satellites,  but  also  from  aeroplanes,  UAV’s,  etc.,  offer 
such  a  source  for  geographical  information. 

The  goal  of  the  study  presented  here  is  to  evaluate  the 
potential  of  remote  sensing  for  geographical 
information  extraction.  The  extracted  information  can 
then  be  used  to  update  outdated  maps  or  to  obtain  basic 
information  about  an  unknown  site. 

For  this  study  we  have  collected  remote  sensing  data 
from  sensors  in  different  wavelength  regions  (optical, 
thermal  infrared  and  microwave)  and  from  different 
platforms  (spaceborne  as  well  as  airborne).  The  data 
have  been  collected  for  two  test  sites  showing  a  variety 
of  geographical  features. 


One  test  site  is  located  in  the  Netherlands  and  shows  no 
significant  relief.  It  is  called  the  ‘Heerde’  test  site  after 
the  Dutch  topographical  map,  which  contains  the  test 
site.  For  this  test  site  a  digital  geographical  database  is 
available,  so  that  a  detailed  comparison  between  the 
remote  sensing  data  and  the  geographical  database  is 
possible. 

A  complete  collection  of  remote  sensing  images 
consisting  of  spaceborne  TM,  SPOT,  KVR,  ERS, 

JERS,  and  airborne  CAESAR  (optical,  multi-spectral), 
PHARS  (microwave  ,  C-band)  and  TIR  data  is 
available. 

The  second  test  site,  located  in  Germany,  near 
Freiburg,  comprises  part  of  the  Rhine  Valley  and  the 
Black  Forest  showing  moderate  relief  up  to  1500  meter. 
For  this  site,  called  the  Freiburg  test  site,  TM,  SPOT, 
ERS,  JERS  and  KVR  data,  including  a  DEM  (DTED) 
are  available.  In  a  following  study  it  will  be  used  to 
investigate  the  influence  of  relief  on  remote  sensing 
images  and  the  generation  of  a  digital  elevation  model 
(DEM)  from  these  images. 


3.  THE  REMOTE  SENSING  DATA 
3.1  Heerde  test  site 

This  test  site  is  located  on  the  TDN  topographical  map 
27.  The  actual  test  site  is  oriented  exactly  east-west, 
and  comprises  an  area  of  10  by  20  km  between  the 
river  IJssel  and  the  Holterberg  to  the  North  of  the  city 
Deventer.  The  location  is  given  by  RD  co-ordinates 
(x,y)  200000, 477500  m  (South  West  corner)  and  (x,y) 
220000,  487500  m  (North  East  corner). 

Low  resolution  satellite  remote  sensing  data  are 
available  for  the  whole  of  map  27  East  and  West  (RD 
180000,475000  (South-west),  RD  220000,500000 
(North-east)),  while  high  resolution  airborne  data  are 
only  available  for  the  actual  test  site. 

The  test  site  includes  the  IJssel  and  its  inundation 
(uiterwaarden)  and  shows  landscapes  like 
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infrastructure,  forest,  agricultural  areas  etc.  The  terrain 
is  flat  up  to  a  few  meters.  For  this  area  the  set  of  images 
shows  resolutions  ranging  from  2  meters  to  30  meters 
enabling  a  detailed  comparison  between  the  ground 
data  and  the  extracted  geographical  information. 

Details  of  the  data  are  given  in  Table  1. 

All  data  were  co-registered  to  the  topographical  maps 
27  East  and  27  West,  with  ground  control  points  and 
using  a  stereographical  projection  and  a  Bessel 
ellipsoid  as  is  usual  for  the  Netherlands.  The  central 
point  of  the  projection  is  located  in  the  city  of 
Amersfoort  (RD  co-ordinates  (x,  y)  155000,  463000  or 
52.2°  lat.,  5.5°  long.).  The  accuracy  of  the  registration 
is  determined  by  the  geometrical  accuracy  of  data  and 


how  accurately  the  ground  control  points  can  be 
determined  with  respect  to  the  topographical  map.  This 
depends  on  the  scale  of  the  maps  and  on  the  resolution 
of  the  data.  For  the  airborne  data  we  used  maps  with  a 
scale  of  1 : 10,000,  so  that  the  data  could  be  registered 
with  an  accuracy  close  to  the  dimension  of  the 
resolution  cell.  (i.e.  3  to  5  m).  The  satellite  data  could 
also  be  registered  with  accuracies  within  the 
dimensions  of  the  resolution  cell  (i.e.  30  -10  m)  using 
maps  of  1:50,000  with  exception  of  the  KVR  data 
which  are  accurately  registered  up  to  10  m,  while  the 
resolution  is  2  m.  The  is  due  to  the  lower  geometrical 
accuracy  of  the  KVR  data. 


Table  1.  Remote  sensing  data  for  the  Heerde  test  area. 


sensor 

platform 

resolution  (m) 

date  (d-m-y) 

TM 

spaceborne 

30 

optical/nir-multispectral 

10-18-1993 

SPOT  XS 

spaceborne 

20 

optical-  multispectral 

10-06-1992 

SPOT  PAN 

spaceborne 

10 

optical-  panchromatic 

13-10-1992 

KVR  1000 

spaceborne 

2 

optical-  photographic 

19-05-1992 

ERS-1 

spaceborne 

25  (3  looks) 

microwave-  C-band/VV/230 

06-08-1992 

JERS-1 

spaceborne 

25  (3  looks) 

microwave-  L-band/HH/35° 

20-09-1993 

CAESAR 

airborne 

3 

optical-  multispectral 

22-09-1994 

PHARS 

airborne 

6  (8  looks) 

microwave  -C-band/VV/45-65° 

27-09-1994 

TIR-camera 

airborne 

5 

thermal  infrared  scanner 

23-09-1994- 

14-10-1994 

PHARS  image  of  part  (4.5  by  3  Km)  of  the  Heerde  test  area 


4.  THE  TERAS  GROUND  DATABASE 

In  order  to  evaluate  the  remote  sensing  data  for 
cartographic  purposes  we  have  used  primarily  the  so- 
called  TERAS  (Terrein  analyse  systeem)  database  [1] 
for  the  Heerde  test  site,  containing  detailed 
geographical  data  in  vector  format. 

These  geographical  data  are  comparable  to  the 
information  on  a  topographical  map  with  scale 
1:50,000. 

At  first  instance  a  collection  of  features  and  attributes 
have  been  selected  on  basis  of  their  relevance  for 
military  terrain  inventory.  These  features  and  attributes 
are  extracted  from  a  DIGEST  (Digital  Geographical 
Information  Exchange  Standard)  code  list  [2].  About 
60  features  are  selected  covering  6  categories,  which 
are  listed  here: 

1 .  Culture;  which  includes  typically  man-made  object 
like  buildings,  roads  etc.  (code  A). 

2.  Hydrography;  for  example  rivers,  canals,  ditches, 
lakes  etc.  (code  B) 


3.  Relief  portrayal;  spot  elevation,  contour  lines 
(code  C) 

4.  Land  forms;  for  example  barren  ground, 
depression  (code  D) 

5.  Vegetation;  for  example  trees,  crop  land  (code  E) 

6.  Demarcation;  e.  g.  an  administrative  boundary 
(code  F) 

Not  all  features  in  the  list  are  suitable  for  comparison 
with  remote  sensing  data.  For  example  administrative 
boundaries  (category  Demarcation)  are  difficult  to 
monitor  with  the  current  remote  sensing  data  or  are  not 
present  in  the  Heerde  test  area.  Some  features,  like 
different  types  of  towers,  were  combined,  since 
distinction  would  involve  identification  which  is  not 
possible  with  the  current  remote  sensing  data  set.  The 
category  relief  portrayal  is  not  considered  here,  but  will 
be  investigated  using  the  second  test  site  near  Freiburg. 
Taking  the  features  suitable  for  comparison  three 
categories  (Culture,  Hydrography  and  Land  forms/ 
Vegetation)  are  left  in  a  compressed  list,  which  is 
shown  below  in  Table  2. 


Table  2  Compressed  list  of  features  used  in  the  comparison  with  remote  sensing  data. 


Culture 


Code 


Hydrography 


code 


Land  forms/  Vegetation 


code 


culvert 

AL240 

tower 

BI030 

lock/weir 

BB140 

jetty 

BH020 

canal 

BI020 

dam 

DA020 

IMIttJtkHI! 

EA010 

cropland 

EB020 

heath 

EA020 
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BH080 

Iake/ponds 

BH090 

inundation 

AL015 

building/com 

plex 

AT040 

pylon 

AL070 

fence 

AN010 

railroad 

AP030 

road 

AT030 

high  tension 
line 

AD030 

substation 

AL020 

build-up  area 

EA040 

orchard 

EC030 

trees 

5.  COMPARISON  REMOTE  SENSING  DATA 
AND  GROUND  DATABASE. 

We  present  here  the  comparison  between  vector  data 
available  in  the  TERAS  data-set  and  remote  sensing 
data  collected  for  this  study.  The  purpose  of  this 
comparison  is  to  evaluate  how  accurately  geographical 
information  can  be  extracted  from  remote  sensing 
sensors  assuming  that  the  TERAS  data-set  can  be  used 
as  a  complete  reference  data-set.  In  this  way  we  try  to 
determine  the  potential  of  remote  sensing  data  for 
geographical  information  extraction. 

In  the  comparison  we  have  used  the  remote  sensing 
images  of  the  Heerde  test  area.  For  the  satellite  images 
we  used  TM,  SPOT-PAN  and  KVR.  The  SPOT-XS 
image  has  been  omitted  in  the  comparison  since  it  is 
expected  that  the  results  will  average  the  results  of 
PAN  and  TM.  Also  the  ERS  and  JERS  images  are  not 
considered  here  since  the  results  are  expected  to  be 
marginal  compared  to  the  other  sensors.  Only  extracted 
point  targets  from  these  images  have  been  compared 
with  point  features  in  the  TERAS  data-base.  These 
results  are  shown  and  discussed  in  section  5.2.  For  the 
airborne  images  we  used  the  CAESAR  data,  PHARS 
data  and  TIR  data. 

By  comparing  the  various  remote  sensing  images  with 
the  TERAS  data-set  (if  needed  also  with  the  digitised 
topographical  map)  we  have  tried  to  determine  for 
every  feature  how  much  of  a  feature  was  seen  (i.e. 
detected)  in  the  image.  To  do  this  different  methods 
had  to  be  used.  For  point  data  it  is  possible  to  count  the 
number  of  points  which  are  clearly  related  to  objects 
seen  in  remote  sensing  images.  The  number  of  points 
which  can  be  evaluated  varies  a  lot  for  the  different 
features.  Usually  the  whole  “Heerde”  test  area  as 
imaged  by  PHARS  and  CAESAR  has  been  taken  for 
evaluation.  In  some  cases  the  number  of  points  is  very 
large  and  a  smaller  area  has  been  taken.  In  case  of  line 
or  polygon  objects  also  the  length  and  area  plays  a  role. 
A  result  is  then  obtained  by  inspection,  for  example  by 
estimating  the  total  detected  length  or  area. 

5.1  Results  of  the  comparison 

The  results  of  the  comparison  are  summarised  in  Table 
3  for  the  six  sensors  mentioned  above.  The  results  are 
shown  for  the  three  classes:  Culture,  Hydrography  and 


Land  forms/  Vegetation.  We  discuss  here  the  results  for 
the  three  classes  separately. 

Culture 

Some  features  like  ‘fences’,  ‘towers’  and  high  tension 
lines  are  generally  too  small  to  be  detected  by  the 
sensors.  For  the  detection  of  these  features  resolutions 
of  less  than  1  meter  are  required  which  are  not 
available  in  the  data-set.  The  low  success  rate  of 
detection  of  towers  is  explained  by  the  fact  that  towers 
are  small  when  they  are  observed  in  vertical  direction. 
For  microwave  sensors  the  viewing  or  look  direction  is 
not  vertical  (slant  range  geometry).  The  detection  of 
towers  is  hindered  in  this  case  since  the  targets  are 
confused  with  other  targets  in  build-up  areas  where 
most  of  the  towers  are  located. 

The  resolution  of  the  TM  sensors  (30  m)  causes  this 
sensor  to  be  less  effective  for  detecting  features  in  the 
category  ‘culture’.  For  SPOT-PAN  the  resolution  of  10 
meter  makes  this  sensor  ‘intermediate’  successful. 
Because  KVR  and  CAESAR  have  relatively  high 
resolutions  (2-3  meter)  most  of  the  features  are 
detected.  Despite  the  somewhat  lower  resolution  of 
CAESAR  (3  m)  compared  to  KVR  (2  m)  the  first  has  a 
slightly  higher  success  rate  than  the  second  because  of 
the  spectral  information.  Due  to  this  information  man¬ 
made  objects  and  vegetation  are  easier  to  discriminate. 
The  results  for  the  TIR  data  are  very  sensitive  to  the 
emission  and  therefore  to  the  temperature  of  the 
objects.  The  TIR  data  used  here  have  been  recorded 
during  quite  optimal  conditions  (late  morning,  clear 
sky).  Man-made  objects  like  bridges,  roads  and 
especially  buildings  are  detected  quite  successfully, 
since  they  have  been  heated  by  solar  radiation. 

For  the  PHARS  sensor  the  smaller  objects  like  culverts 
and  the  smaller  roads  are  difficult  to  detect  because  of 
the  relatively  low  resolution  (6  m)  combined  with  the 
speckle. 

Hydrography 

The  dimension  of  lakes  and  canals  can  vary 
substantially.  For  example  the  feature  ‘lakes’  in  the 
TERAS  data-base  contains  many  ponds  sometimes 
overgrown  by  trees  and  the  feature  ‘canals’  contains 
many  small  ditches.  This  causes  the  detection  rate  to  be 
quite  low  for  the  lower  resolution  sensors  (TM  and 
SPOT-PAN)  despite  the  fact  that  these  features  are 
generally  easy  to  detect. 
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The  feature  ‘inundation’  (IJssel  Uiterwaarden)  is 
difficult  to  observe  directly  since  its  main  property  is 
small  scale  relief  over  a  large  area.  However  the  land 
use  is  different  for  the  inundation  compared  to  the 
surroundings  so  that  observation  of  the  land  use  makes 
detection  indirectly  possible.  It  appears  that  high 
resolution  (<  5  meter)  is  advantageous  for  this  purpose. 
The  discrimination  between  water  and  vegetation  is 
clearly  more  difficult  for  PAN  recordings  (SPOT, 

KVR)  compared  to  multi-spectral  recordings 
(CAESAR). 

For  the  PHARS  sensor  the  detection  of  canals  is 
hindered  by  confusion  with  roads. 

Despite  the  relatively  high  resolution  of  the  TIR  sensor 
(5  meter)  the  detection  rate  is  substantially  lower  than 
that  for  the  other  high  resolution  systems  (KVR, 
CAESAR)  due  to  the  fact  that  the  contrast  between 
water  and  the  surroundings  is  not  very  high,  especially 
when  the  water  is  surrounded  by  vegetation.  However  it 
should  be  noted  that  under  certain  circumstances,  e.g. 
relatively  warm  water  on  a  cold  night  water  can  be  a 
dominant  feature  in  the  image. 

Land  forms/  Vegetation 

In  this  case  resolution  is  a  less  crucial  parameter,  since 
most  objects  are  extended.  For  detection  other 


information  is  important.  Especially  multi-spectral 
information  plays  a  dominant  role.  This  is  due  to  the 
fact  that  vegetation  reflects  most  of  the  light  in  the 
near-infrared  compared  to  e.g.  man-made  objects. 

By  observing  in  the  near-infrared  vegetation 
differences  also  become  apparent. 

It  is  therefore  that  the  TM  sensor  has  an  even  higher 
detection  rate  compared  to  the  SPOT-PAN  sensor 
despite  the  difference  in  resolution.  The  same  is  true 
for  the  CAESAR  compared  to  the  KVR  sensor. 

The  detection  rate  of  the  PHARS  sensors  is  lower  on 
average  compared  to  the  multi-spectral  optical  sensors. 
Like  in  the  optical,  also  in  the  microwave  wavelength 
region  vegetation  differences  are  more  easily  observed 
when  additional  information  is  available,  for  example 
from  multi-wavelength  systems  or  polarimetric 
systems.  The  single  channel  PHARS  sensor  therefore 
shows  a  relatively  low  detection  rate. 

The  TIR  sensor  is  not  ideal  to  discriminate  between 
different  types  of  vegetation,  since  vegetation  attempts 
to  suppress  temperature  differences  by  controlling  the 
evaporation  of  plant  moisture.  The  detection  rate  is 
consequently  relatively  low. 


Table  3.  Summary  of  the  ability  to  detect  features  for  the  different  sensors  _ 

|  I  Name  I  TM  I  PAN  |  KVR  I  CAESAR  I  PHARS  |  TIR 


Results  for  category  culture. 


AQ040 

10% 

50% 

80% 

>90% 

50% 

90% 

AO065 

culvert 

<  10% 

20% 

70% 

80% 

20% 

20% 

"1  ^ 

0% 

30% 

90% 

90% 

10% 

50% 

AL240 

tower 

<  10% 

<  10% 

10% 

10% 

<  10% 

<  10% 

AL015 

building/com 

plex 

50% 

90% 

100% 

>  90% 

80% 

100% 

AT040 

pylon 

<  10% 

10% 

70% 

90% 

90% 

<  10% 

AL070 

fence 

0% 

0% 

<  10% 

<  10% 

0% 

0% 

AN010 

railroad 

70% 

90% 

100% 

100% 

100% 

80% 

AP030 

road 

20% 

70% 

>90% 

100% 

50% 

70% 

AT030 

high  tension 
line 

0% 

0% 

0% 

0% 

<  10% 

0% 

AD030 

substation 

<  10% 

60% 

90% 

90% 

90% 

90% 

AL020 

BwnaiBw^ii 

60% 

80% 

>90% 

100% 

>90% 

100% 

|  Culture  |  Average  |  21%  |  43%  |  67%  I  71%  |  50%  |  52% 


Results  for  category  hydrography 


BI030 

D — tl  _  P. 

lock/weir 

1  .V. - 

0% 

0% 

30% 

40% 

40% 

40% 

BB140 

l'etty 

<  10% 

40% 

90% 

>90% 

>90% 

50% 

BH020 

canal 

30% 

20% 

60% 

80% 

30% 

50% 

BI020 

dam 

20% 

40% 

50% 

40% 

40% 

30% 

BH080 

30% 

30% 

60% 

80% 

80% 

50% 

BH090 

inundation 

0% 

30% 

50% 

50% 

30% 

50% 

|  Hydrography  |  Average 


15% 


|  27%  |  57%  1  63% 


1  52%  |  45% 
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Results  for  category  land  forms/vegetation 


DA020 

90% 

70% 

70% 

100% 

20% 

20% 

EA010 

60% 

60% 

70% 

90% 

80% 

70% 

EB020 

heath 

70% 

80% 

80% 

80% 

50% 

40% 

EA020 

50% 

70% 

80% 

90% 

80% 

90% 

EA040 

orchard 

70% 

10% 

50% 

50% 

20% 

<  10% 

EC030 

trees 

60% 

80% 

90% 

100% 

90% 

100% 

Land  forms/ 

Average 

67% 

62% 

73% 

85% 

57% 

55%  i 

vegetation 


5.2  Other  comparisons  with  remote  sensing  data 

Roads 

By  displaying  the  remote  sensing  images  on  screen  and 
digitising  the  detected  roads  (AP030)  manually  a 
database  was  created  containing  vector  data  for  the 
different  remote  sensing  sensors  for  the  Heerde  test 
area.  The  total  length  of  the  roads  can  be  calculated  and 
compared  with  the  total  length  of  the  roads  in  the 
TERAS  database.  In  this  way  a  more  quantitative 
comparison  can  be  obtained  compared  with  the 
inspection  method  discussed  above.  We  show  the  total 
length  (in  meter)  and  some  statistics  in  Table  4. 

Note  that  for  PHARS  two  entries  are  available  in  Table 
4.  For  PHARS  I  only  the  more  obvious  cases  (dark 
lines)  are  extracted  as  roads.  For  PHARS  II  less 
restrictions  are  made  and  most  of  the  dark  lines  are 
extracted  as  roads.  Since  roads  (AP030)  and  canals 
(BH020)  are  easily  confused  in  microwave  images  the 
result  will  be  overestimated. 

Point  features 

Microwave  images  are  suitable  for  automatic  point 
target  extraction.  Point  targets  can  give  increased 
backscatter  which  can  easily  be  discriminated  from  the 
background  with  statistical  means.  The  extracted  points 
can  be  compared  automatically  with  point  features  in 
the  TERAS  data-base,  so  that  a  quantitative 
comparison  is  possible.  In  this  way  ERS  and  JERS 


data,  having  improperly  low  resolutions  for 
cartographic  applications,  may  be  used  for  the 
detection  of  point  features.  The  comparison  has  been 
made  for  three  ERS  and  one  JERS  images  (see  Table  5) 
and  for  the  complete  area  covered  by  topographical 
maps  27  East  and  27  West.  A  point  was  extracted  from 
the  images  when  the  backscatter  was  4.3  dB  above  the 
background  giving  more  than  99%  confidence  that  the 
point  is  not  due  to  speckle. 

About  3500  points  representing  mostly  buildings  and 
towers  are  selected  from  the  TERAS  database.  A  point 
feature  in  the  TERAS  data-set  selection  was  said  to  be 
detected  when  it  coincided  within  50  m  with  a  point 
extracted  from  the  RS  image.  The  result  is  that  about 
20%  of  the  point  features  in  the  TERAS  data-base  are 
‘seen’  by  ERS  and  JERS  (see  Table).  Another  question 
is  then  how  many  of  the  ‘detected’  points  are  the  same 
for  all  four  cases,  since  the  data  differ  not  only  in  time 
but  also  in  azimuth  angle  (different  for  descending  and 
ascending  passes)  and  in  sensor  (ERS/JERS).  It  appears 
that  for  cases  1  and  2  (difference  in  azimuth  angle  and 
time)  40%  is  the  same,  while  for  cases  2  and  3  70%  is 
the  same  (difference  in  time  only)  and  for  cases  3  and  4 
30%  is  the  same  (difference  in  sensor  and  time).  The 
low  value  for  cases  1  and  2  can  be  explained  by  the  fact 
that  the  backscatter  from  point  targets  is  quite 
dependent  on  the  azimuth  angle. 


Table  4.  Statistical  data  for  AP030  (roads). 


database 

no.  ofelem. 

total  length  (m) 

mean  length  (m) 

s.d.  length  (m) 

TERAS 

1646 

433695 

263 

242 

TM 

1 

107505 

- 

- 

PAN 

169 

311750 

1845 

1621 

KVR 

2536 

453285 

178 

201 

CAESAR 

807 

619553 

768 

1256 

PHARS I 

64 

147523 

2305 

2876 

PHARS  II 

311 

417945 

1344 

1638 

TOP50 

702 

557823 

774 

1337 

Table  5.  Comparison  results  for  the  ERS/JERS  and  TERAS  point  data-set 


Case 

Data 

Date 

Pass 

Result 

1 

ERS  1 

23  06  92 

ascending 

22% 

2 

ERS  2 

02  07  92 

descending 

20% 

3 

ERS  3 

23  06  92 

descending 

24% 

4 

JERS 

20  09  93 

descending 

20% 
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6.  CONCLUSIONS 

It  is  difficult  to  draw  general  conclusions  for  the  whole 
set  of  features,  since  the  characteristics  of  the  features 
are  quite  diverse.  In  principle  every  feature  needs  to  be 
considered  on  its  own.  By  grouping  features  into 
classes  some  general  conclusions  can  be  drawn,  since 
many  of  the  ‘culture’  features  are  point  targets,  while 
the  ‘land  forms/  vegetation’  features  are  often 
polygons. 

In  general  we  can  say  that  most  of  the  features  of  the 
DIGEST  list  in  Table  2  can  be  detected  and  recognised 
when  the  resolution  and  the  wavelength  is  appropriately 
chosen.  Man-made  features  like  building  or  roads  can 
be  detected  and  often  recognised  quite  adequately  when 
the  resolution  is  appropriate,  i.e.  less  than  5  meter.  The 
recognition  of  towers  is  difficult  when  the  viewing 
direction  is  vertical.  Direct  detection  of  fences  and 
power  transmission  lines  is  difficult  due  to  their  small 
dimensions.  For  resolutions  less  than  1  meter  most 
features  can  be  recognised.  For  extended  features  like 
crop  land  or  forest  multi-spectral  or  polarimetric 
information  is  more  important  than  the  resolution. 

Some  features  like  inundation  (BH090)  and 
underground  water  (BH1 15)  are  difficult  to  observe 
directly  with  remote  sensing  sensors.  Sometimes 
information  can  be  obtained  indirectly  by  observing 
other  features  which  are  related  (i.e.  context 
information;  for  example,  green  vegetation  in  case  of 
available  underground  water). 

Optical  sensors  make  use  of  daylight  and  are  weather 
dependent.  For  optical  sensors  high  resolution  images 
can  be  obtained,  which  facilitates  the  recognition  of 
objects.  Radiometric  resolution  and  spectral 
information  are  also  quite  important  to  discriminate 
objects,  especially  between  made-made  objects  and 
vegetation.  Near-infrared  information  is  very  useful  to 
discriminate  different  types  of  vegetation. 

TIR  sensors  are  very  suitable  to  detect  and  to  recognise 
man-made  objects,  like  buildings,  roads  etc.,  when  the 
circumstances  are  good  (i.e.  clear  and  sunny  weather). 
However  the  circumstances  are  very  crucial  for  TIR 
sensors.  During  a  cloudy,  misty  or  rainy  day  the 
contrast  vanishes  and  objects  (also  man-made  objects) 
are  hardly  visible  in  the  image.  For  the  discrimination 
of  vegetation  types  TIR  sensors  are  less  suitable. 

The  main  advantage  of  microwave  sensors  is  that  they 
are  independent  of  most  circumstances  since  it  is  an 
active  sensor  which  can  be  used  in  principle  during  all 
weather  conditions  and  at  night.  Microwave  sensors 
can  be  appropriate  for  detecting  objects  due  to  specular 
reflections  which  give  high  returns  in  the  image  even 
when  the  resolution  Cell  is  significantly  larger  than  the 
object  (e.g.  high  tension  pylons  in  ERS  images).  The 
result  is  a  point  feature  which  does  not  allow 
recognition.  In  general  the  ability  of  microwave  sensors 
to  recognise  objects  is  rather  low.  Even  when  more 


resolution  cells  are  covering  the  object  recognition  is 
difficult  since  then  usually  only  a  collection  of  points  is 
seen.  The  contour  of  the  object  which  is  the  main 
feature  for  recognition  in  the  optical  and  TIR  is  in 
microwave  image  suppressed  due  the  appearance  of 
specular  reflections  and  speckle.  The  low  resolution  of 
present-day  spaceborne  sensors  is  clearly 
disadvantageous  for  obtaining  geographical 
information. 

A  quite  important  concept  for  detection  and  recognition 
is  ‘context  information’.  The  context  can  provide 
information  about  a  feature  even  when  the  feature 
cannot  be  observed  directly.  Since  in  many  cases 
features  show  up  as  small  objects  in  the  image  only  a 
few  pixels  provide  information  limiting  the  recognition. 
However  other  independent  information  available  for 
the  interpreter,  for  example  background  information 
about  location  and  the  surroundings  can  provide  the 
essential  information  so  that  recognition  or  even 
identification  becomes  possible.  This  so-called  ‘context 
information’  will  always  play  an  important  role  in  using 
remote  sensing  images. 
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PAPER  No:  3 

DISCUSSOR'S  NAME:  G.  S.  Brown 

COMMENT/QUESTION: 

What  is  your  experience  using  radar  data  over  foliage  covered  terrain? 


AUTHOR/PRESENTER’S  REPLY: 

Foliage  coverage  from  forests  and  tree-lines  can  be  detected  by  radar,  but  the  polarimetric  radar 
performs  much  better  in  detection  and  classification  of  trees  and  tree  types  through  their  foliage 
than  the  single  channel,  fixed  polarization.  For  the  detection  of  structures  under  the  foliage, 
penetration  is  required.  Foliage  penetration  is  only  possible  with  low  frequency  radars,  which  are 
usually  limited  in  their  resolution. 


4-1 


Design  of  a  Multi-sensor  System  for  3D  Site  Model 
Acquisition  and  Exploitation 


I.  TANNOUS*,  S.  GOBERT*,  T.  LAUREN£OT*,  J.-M.  DULAC",  O.  GORETTA*** 


*THOMSON-CSF/  SYSECA 
Image  and  Geographic  Information  Unit 
R&D  Department 
66,  avenue  Pierre  Brossolette 
92247  Malakoff  -  FRANCE 
Tel:  33-01-41-48-03-84 
Fax:  33-01-41-48-02-54 
E-mail:  issam.tannous@syseca.thomson.fr 


"THOMSON  TRAINING  &  SIMULATION 
Tool  &  Database  Product 
5,  rue  Leonardo  Da  Vinci,  BP  252 
78703  Conflans  Ste  Honorine 
FRANCE 

Tel:  33-01-34-90-35-99 
Fax:  33-01-34-90-36-02 


‘“DGA/DSP/SPOTl/OER 
1 8,  rue  du  Dr  Zamenhof 
92131  Issy-les-Moulineaux 
FRANCE 

Tel:  33-01-41-46-23-03 
Fax:  33-01-41-46-33-04 


ABSTRACT 

The  increasing  number  of  high  resolution  image  sensors  in  a  near  future,  as  well  as  the  need  for  realistic  3D  site 
databases  for  applications  like  simulation,  terrestrial  mission  preparation  and  training,  has  emphasized  the  need  for 
multi-sensor  image-based  system  dedicated  to  the  production  and  exploitation  of  3D  site  model.  We  present  here  the 
design  of  such  a  system,  including  the  required  functionnalities  for  this  purpose.  The  system  comprises  a  geometric 
reconstruction  step,  which  aims  at  producing  a  geometrically  reliable  3D  model  of  the  site,  including  the  topography, 
from  the  aerial  or  satellite  images  available.  Next  step  is  the  exploitation  of  other  images,  especially  terrestrial 
photographs,  for  the  mapping  of  real  textures  on  the  faces  of  the  reconstucted  objects,  as  well  as  the  texturation  of 
the  terrain.  The  output  of  this  stage  is  a  realistic  3D  site  model  that  can  be  used  by  the  exploitation  module  of  the 
system,  allowing  the  real  time  reconstruction  of  the  scene  from  any  point  of  view,  interactively  or  according  to  a  pre¬ 
defined  scenario.  This  system  has  been  prototyped  by  SYSECA  and  THOMSON  TRAINING  &  SIMULATION 
(both  subsidiaries  of  THOMSON-CSF)  using  their  own  software,  and  the  results  of  an  experimentation  with  real 
data  have  demonstrated  the  feasibility  of  our  approach. 


INTRODUCTION 

The  availability  of  an  increasing  number  of  high  resolution  im¬ 
aging  sensors  (airborne  and  spaceborne)  emphasizes  the  new 
capabilities  of  image-based  systems  for  a  lot  of  military  (as  well 
as  civilian)  applications  relying  on  this  means  of  observation.  In 
addition  to  better  coverage  and  availability  of  the  image  data 
due  to  their  increased  number,  the  high  resolution  of  these  sen¬ 
sors  allows  the  characterization  and  reconstruction  of  human¬ 
sized  objects  from  the  images  ;  with  the  1 -meter  class  imaging 
satellites  recently  launched  (or  to  be  launched  soon,  see  [2]),  it 
will  be  possible  to  reconstruct  an  entire  3D  site  model  from 
space,  including  the  shape  and  description  of  buildings,  roads 
and  environment  (trees,  forests,  relief). 

The  applications  which  benefit  from  these  capabilities  are  typi¬ 
cally  the  surveillance,  monitoring  and  change  detection  on  a  site 
of  interest,  as  well  as  terrestrial  mission  repetition  and  training. 
Such  applications  are  based  on  the  exploitation  of  3D  site  mod¬ 
els,  and  their  efficiency  rely  on  the  quality  of  these  models.  The 
type  of  3D  site  modelling  needed  for  these  applications  must 
cope  with  two  kinds  of  need  : 

—  the  site  model  must  be  geometrically  reliable,  which  means 
that  the  location  and  dimensions  of  the  objects  must  be  close 
enough  to  the  real  ones  ;  such  a  constraint  is  emphasized  by 
the  need  for  high  level  of  details  required  in  applications  like 
terrestrial  mission  preparation  ; 

—  the  site  model  must  be  photometrically  reliable,  which 
means  that  the  model  must  include  a  realistic  representation 
of  the  appearance  of  the  faces  (textures)  of  the  objects  of  the 
site. 


With  these  constraints,  it  appears  that  a  single  source  of  images 
is  not  sufficient :  it  is  clear  that  a  single  image  cannot  lead  to  a 
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reliable  3D  reconstruction  of  a  site  ;  futhermore,  even  if  several 
images  of  a  same  sensor  are  exploited  in  stereoscopy  for  the 
geometric  reconstruction,  the  angles  of  view  cannot  vary 
enough  to  give  an  exhaustive  view  of  the  site  ;  moreover,  a  sin¬ 
gle  sensor  offers  generally  one  single  spectral  band,  which  is  not 
sufficient  for  a  realistic  site  reconstruction  in  terms  of  photom¬ 
etry.  That’s  why  it  is  necessary  to  acquire  images  from  various 
sensors,  from  various  angles  of  view,  to  be  able  to  reconstruct  a 
reliable  3D  model  site. 

These  considerations  lay  the  stress  on  the  fact  that  a  3D  site 
model  acquisition  system  must  offer  multi-sensor  capabilities, 
insuring  the  optimal  use  of  this  amount  of  data  for  3D  informa¬ 
tion  extraction.  Consequently,  an  exploitation  system  must  be 
able  to  cope  with  these  complex  data  sets,  including  3D  geom¬ 
etry  and  textures. 

Our  paper  deals  with  the  design  and  experimentation  of  a  sys¬ 
tem  allowing  the  acquisition  and  updating  of  3D  models  of  sites 
of  interest  using  sources  like  high  resolution  imaging  satellites, 
aircrafts  or  UAVs  cameras,  and  terrestrial  photographies  (see 
figure  1).  The  data  provided  by  this  system  has  been  formatted 
to  be  exploited  by  existant  software  package  dedicated  to  train¬ 
ing  and  simulation.  As  a  result,  we  demonstrated  the  feasibility 
of  this  approach  and  we  validated  our  site  model  acquisition 
system  design. 

This  paper  is  organized  in  4  parts.  First,  we  describe  the  work- 
flow  of  3D  site  model  acquisition  and  exploitation,  determining 
three  phases  :  the  geometric  reconstruction,  the  rendering,  and 
the  exploitation.  The  second  part  describes  in  details  the  geo¬ 
metric  reconstruction  module.  Then  we  focus  on  the  rendering 
step.  Next,  we  describe  quickly  the  exploitation.  Finally,  we 
present  the  results  obtained  with  our  prototype. 


1.  DESCRIPTION  OF  THE  SYSTEM’S  WORKFLOW 

Our  system  is  divided  in  modules,  and  the  typical  workflow  for 
3D  site  modelling  is  represented  on  figure  2.  The  three  main 
modules  of  the  system  are  the  Geometric  Reconstruction  mod¬ 
ule  ,  the  Rendering  module,  and  the  Exploitation  module. 

1.1.  Geometric  Reconstruction 

The  geometric  reconstruction  is  based  on  photogrammetry  tech¬ 
niques,  where  one  can  estimate  the  true  geometry  of  the  real  ob¬ 
jects  from  the  images  obtained  with  a  sensor  whose  geometric 
configuration  is  known  (the  so-called  geometric  model  of  the 
images).  In  this  step,  the  images  that  can  be  used  are  generally 
satellite  or  aerial  images,  because  the  photogrammetric  exploi¬ 
tation  of  such  sensors  is  possible,  whereas  terrestrial  photogra¬ 
phies  are  often  unexploitable  in  this  way. 

The  first  step  consists  in  a  geometric  pre-processing,  which  is 
compulsory  for  the  3D  model  extraction  from  the  images.  It 
comprises  the  so-called  aero/spatio-triangulation,  which  aims 
at  computing  the  geometric  orientation  parameters  of  each  im¬ 
age,  coherently  with  the  other  images.  These  parameters  are 
necessary  to  be  able  to  infer  real  geometric  information  from  the 
images,  while  establishing  the  exact  geometric  correspondance 
between  images  objects  and  real-world  objects.  This  module  is 
based  on  an  original  multi-sensor  triangulation  algorithm  devel¬ 
oped  by  SYSECA,  allowing  the  coherent  geometric  registration 
of  heterogeneous  images  (e.g.  SPOT  and  aerial  images).  In  this 
step,  a  DTM  (Digital  Terrain  Model)  and  an  orthophoto  are  gen¬ 


erated  from  the  images,  using  an  original  automatic  algorithm 
developed  by  SYSECA. 


images  ancillary  data  georeferenced  data 


Figure  2  :  System’s  workflow 

Next  step  is  the  3D  models  extraction ,  where  the  system  pro¬ 
vides  the  tools  for  modelling  the  objects  of  the  site  from  their 
views  in  the  images.  It  includes  two  displays:  one  for  the  control 
of  the  tools  and  the  visualization  of  the  database  that  is  being 
constructed,  and  another  for  the  extraction  of  the  objects  from 
the  images.  The  second  display  is  a  stereoscopic  display,  allow¬ 
ing  the  stereoscopic  viewing  of  pair  of  images.  In  case  of  heter¬ 
ogenous  images,  this  display  is  used  in  a  split-screen  mode. 
During  this  step,  3D  models  of  the  objects  are  extracted  and 
stored  in  the  3D  DataBase.  This  step  uses  an  original  software 
module  developed  by  SYSECA  (see  [9]). 

In  order  to  allow  the  introduction  of  more  complex  objects  in 
the  3D  data  base,  or  the  addition  of  geometric  details  not  ex¬ 
tracted  from  the  images  (chemineas  or  balconies),  the  extraction 
module  comprises  a  CAD-modeller  environment,  where  the 
user  can  build  complex  objects  from  the  objects  of  the  database, 
add  others  objects  on  existing  objects,  or  create  new  objects. 
This  step  can  be  used  for  entering  known  dimensions  of  objects 
(e.g.  extracted  from  plans).  The  software  used  for  this  step  is 
Ideal 5  from  THOMSON  TRAINING  &  SIMULATION 
(TT&S). 

1.2.  Rendering 

After  having  reconstructed  a  geometrically  reliable  3D  site 
model,  the  photometric  reconstruction  can  be  done.  In  this  step, 
we  map  the  textures  extracted  from  the  images  on  the  faces  of 
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the  extracted  geometric  model.  We  make  the  distinction  be¬ 
tween  the  mapping  of  textures  from  terrestrial  images,  which 
must  be  done  interactively,  and  the  mapping  of  textures  from 
the  photogrammetrically  exploited  images  (aerial  or  satellite). 

We  call  the  manual  process  of  texturation  from  terrestrial  imag¬ 
es  the  Texture  Mapping ,  and  the  step  where  textures  are  mapped 
from  registered  images  is  called  the  Terrain  Integration.  These 
steps  are  described  with  more  details  on  section  3.  It  is  based  on 
a  classical  image  software  like  Photoshop  for  the  manual  ex¬ 
traction  of  textures  from  terrestrial  images,  and  on  the  TT&S 
software  ModTer  for  the  Terrain  Integration.  In  addition,  the 
Terrain  Integration  is  in  charge  of  the  formatting  of  the  database 
and  the  DTM  for  the  visualisation  (triangulation). 

1.3.  Exploitation 

Final  step  is  the  exploitation  of  this  3D  model  site.  This  exploi¬ 
tation  consists  in  reconstructing  a  realistic  view  of  the  site  in 
real  time  from  any  point  of  view.  Such  realistic  simulation  can 
be  performed  using  several  classes  of  observers  or  vehicles, 
whose  behaviour  can  be  accounted  for  in  the  deplacement 
through  the  site  (pedestrian,  tank,  helicopter,  aircraft, ...). 

This  exploitation  comprises  a  preparation  step,  where  a  scenario 
can  be  defined.  Then,  the  scenario  is  played  in  the  Animation 
step.  During  the  animation,  the  user  is  able  to  play  a  recorded 
trajectory,  or  interact  in  real  time  to  move  freely  around  the  site. 

The  prototype  is  based  on  two  software  products  from  TT&S  : 
TrackMaker  for  the  scenario  definition,  and  SpaceMagic  for  the 
Animation. 

In  the  following,  we  describe  more  precisely  the  key  parts  of  our 
system. 


2.  GEOMETRIC  RECONSTRUCTION 

This  section  decribes  more  precisely  the  geometric  reconstruc¬ 
tion  step.  After  describing  the  geometric  processing,  we  focus 
on  the  critical  aspects  of  this  step.  We  will  insist  on  the  problem 
of  management  of  3D  data,  and  the  solution  we  propose.  Then, 
the  extraction  environment  will  be  detailed,  with  the  description 
of  the  Machine  Man  Interface  configuration,  the  3D  extraction 
models  and  their  acquisition  modes,  and  finally  the  semi-auto¬ 
matic  tools  we  implemented  to  support  the  extraction. 

2.1.  Geometric  pre-processing 

The  aero/spatio-triangulation  process  is  based  on  Ground  Con¬ 
trol  Points  introduced  interactively  in  the  system,  on  the  images, 
and  tie-points  between  couples  of  images  found  automatically, 
as  well  as  the  ancillary  data  provided  with  the  images  (ephemer- 
ids,  interior  ortientation  of  the  sensor, ...).  The  ancillary  data  are 
used  for  automatically  determining  the  interior  orientation  of 
the  image,  and  to  provide  initial  exterior  orientation  parameters 
for  the  triangulation.  The  exact  parameters  are  then  estimated 
using  classical  photogrammetric  techniques  (least-squares  ad¬ 
justments).  The  algorithm  is  fully  multi-sensor,  which  means 
the  coherent  registration  of  many  heterogeneous  images  (satel¬ 
lites  and  aerial)  is  possible. 

The  generation  of  the  DTM  is  done  automatically  using  the  im¬ 
ages.  This  DTM  is  computed  on  the  whole  area  covered  by  the 
images  (where  at  least  two  images  are  available).  In  order  to 
prevent  from  the  limitations  of  the  correlation-based  DTM  gen- 
eratio  algorithm,  it  includes  an  erosion  step,  where  the  above¬ 
ground  (artefacts  due  to  superstructures)  is  automatically  re¬ 


moved.  This  automatic  step  can  be  followed  by  an  interactive 
erosion  step  for  the  residual  above-ground. 

The  generation  of  the  orthophoto  is  performed  by  using  the  ori¬ 
entation  parameters  of  the  images  and  the  DTM  to  compute  a 
geometrically  rectified  image  (geocoded),  where  the  effects  of 
the  sensor  projection  and  the  terrain  induced  deformations  are 
removed.  This  orthophoto  covers  the  whole  imaged  site  by  mo¬ 
saicking  the  images. 

2.2.  Management  of  three  dimensional  data 

The  3D  model  of  a  site  is  composed  of  many  3D  primitives, 
which  must  be  stored  and  managed  specifically  to  be  exploit¬ 
able  by  each  module  of  the  system.  Moreover,  the  coherence  of 
the  data  seen  by  different  modules  must  be  insured.  Such  spec¬ 
ifications  show  that  the  3D  data  management  problem  in  such  a 
system  has  to  be  studied  carefully. 

Management  of  three  dimensional  data  is  traditionally  a  CAD 
world  concern,  whereas  management  of  geographic  data  is  a 
GIS  domain  concern.  Though,  current  commercial  GIS  are  not 
able  to  cope  with  3D  objects.  The  geographic  data  are  essential¬ 
ly  planimetric,  with  a  z  coordinate.  The  resulting  dimension  is 
called  2D5.  A  fully  3D  data  model  is  not  implemented  in  GIS, 
because  of  the  high  complexity  of  3D  topology. 


Figure  3  :  Management  of  3D  data  with  GIS  and  CAD  Modeller 


Instead  of  trying  to  develop  a  heavy  fully  3D  GIS,  we  prefered 
to  adopt  a  composite  data  model,  taking  the  best  of  both  tech¬ 
nologies.  The  choice  of  this  design  is  driven  by  the  fact  that  the 
3D-space  indexing,  necessary  for  some  applications  like  geo¬ 
logical  survey  (see  reference  [8]),  is  not  essential  for  our  appli¬ 
cation,  which  is  dedicated  to  the  above-ground  extraction.  So, 
the  cartographic  approach  offered  by  a  GIS  is  still  relevant.  To 
introduce  the  third  dimension  in  the  planimetric  objects  of  a 
2D5  GIS,  the  solution  is  to  associate  a  3D  description  to  the 
footprint  symbolizing  a  building  in  the  GIS.  Each  building  is 
then  stored  as  a  footprint  associated  with  a  CAD  description 
(see  figure  3).  The  association  can  be  implemented  with  a  Rela¬ 
tional  DBMS.  Our  approach,  close  to  the  one  proposed  by 
Forstner  [1],  offers  a  powerful  solution  at  a  moderate  cost  for 
development  of  3D  GIS  devoted  to  3D  site  modelling.  Futher- 
more,  the  CAD  approach  for  the  3D  description  of  the  objects 
allows  a  high  level  of  detail,  and  can  include  a  photorealistic 
texture  for  each  facet  of  the  objects. 

2.3.  Configuration  of  the  extraction 

The  design  of  the  Man-Machine-Interface  dedicated  to  3D  mod¬ 
el  extraction  from  images  is  a  difficult  task,  and  our  approach  is 
the  result  of  many  concertations  with  operational  staff. 
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Two  displays  are  used: 

—  the  monitor  screen  is  a  classic  monitor,  for  controlling  the 
interactive  tools  and  the  database  that  is  being  reconstructed, 

—  the  exploitation  screen,  for  viewing  the  images,  is  able  to 
display  the  images  in  monoscopic  and  stereoscopic  mode. 
The  3D  objects  that  are  being  extracted  are  viewed  as  wire¬ 
frame  models  superimposed  on  the  images. 

For  the  nominal  use  of  the  system,  we  suppose  a  stereoscopic 
pair  of  images  is  available  and  is  viewed  on  the  exploitation 
screen. 

In  order  to  view  and  control  the  database,  there  are  two  specific 
windows  on  the  monitor  screen: 

—  a  cartographic  window,  where  the  footprints  of  the  objects 
are  symbolized  (similarly  as  in  a  GIS  window);  this  window 
offers  a  synthetic  view  of  the  map  of  the  site,  showing  the 
areas  that  have  been  reconstructed;  it  can  be  also  useful  for 
comparison  of  the  reconstruction  to  an  existing  map. 

—  a  perspective  window,  where  the  3D  objects  of  the  site  are 
viewed  in  perspective,  from  a  point  of  view  which  can  be  in¬ 
teractively  moved;  in  this  window  it  is  possible  to  view  si¬ 
multaneously  the  DTM  as  a  wireframe;  futhermore,  it  is  also 
possible  to  drape  the  orthophoto  on  the  DTM,  which  offers  a 
good  realism  to  the  operator. 

The  three  kinds  of  visualization  of  the  database  (exploitation 
screen,  cartographic  window  and  perspective  window)  are  co- 
herentely  managed,  which  means  every  action  on  the  objects 
from  one  window  is  taken  into  account  by  the  others. 

2.4.  3D  Models 

The  type  of  objects  to  be  extracted  are  divided  in  three  classes: 
the  planimetric  objects  (roads,  rivers,  areas,  parkings,  ...),  the 
volumetric  objects  (buildings),  and  the  generic  objects  (pylons, 
roadsigns,  trees,  antennas,...). 

The  planimetric  objects  use  classic  2D5  primitives,  like 
polylines,  ribbons,  or  polygons.  They  are  managed  as  usual  in 
the  GIS. 


I  extruded  polygon 

Figure  4  :  Volumetric  primitives 

The  volumetric  objects  entailed  a  specific  process.  Among  the 
various  approaches  of  3D  modelling  in  the  CAD  world,  we  de¬ 
cided  to  focus  on  Constructive  Solid  Geometry,  where  the  only 
graphic  primitives  are  volumetric.  Objects  are  defined  as  a  com¬ 
position  of  elementary  volumetric  primitives.  This  ensures  that 
the  volumetric  objects  of  thcJD  database  are  geometrically  co¬ 
herent.  The  primitives  used  for  volumetric  objects  acquisition 
can  be  defined  by  the  users,  and  an  example  of  such  objects  is 
given  on  figure  4. 

The  extraction  is  done  from  the  images  on  the  stereoscopic  dis¬ 
play.  Each  type  of  object  implies  an  acquisition  protocole.  For 
example,  a  parallelepiped  is  extracted  by  four  (3D)  points  to  de¬ 
fine  the  roof,  and  a  pointing  on  the  ground  to  define  the  altitude 
of  the  basis  of  the  building.  A  peaked  roof  house  can  be  extract¬ 
ed  by  only  five  points  and  a  point  on  the  ground.  The  selection 


of  the  type  of  object  to  be  extracted  is  done  through  an  ergo¬ 
nomic  man-machine  interface.  Some  examples  of  primitives  ac¬ 
quisition  are  given  on  figure  5. 

The  generic  objects  (pylons, ...)  are  too  complex  to  be  fully  ex¬ 
tracted,  but  they  can  be  represented  by  a  small  number  of  gener¬ 
ic  model,  common  to  almost  every  site.  Our  approach  is  to 
extract  from  the  images  only  the  3D  bounding  box  of  the  ob¬ 
jects,  whose  CAD  model  is  chosen  in  a  generic  model  database. 
As  a  result,  the  extraction  task  is  simple  whereas  the  graphic 
model  is  highly  complex  and  realistic. 


2.5.  Semi-automatic  extraction  tools 

In  order  to  increase  the  productivity  of  the  extraction,  several 
tools  are  available  in  order  to  ease  the  acquisition  of  the  objects. 
Because  of  the  lack  of  maturity  of  most  fully  automatic  extrac¬ 
tion  tools,  we  focus  on  semi-automatic  extraction  tools.  The  ini¬ 
tial  positioning  of  the  primitives  to  be  extracted  is  performed  by 
the  user,  and  the  fine  adjustment  of  the  primitive  to  the  images 
is  in  charge  of  the  computer.  The  semi-automatic  approach  is 
probably  the  most  promising  way  of  increasing  the  productivity 
extraction  by  means  of  image  processing  algorithms  (see  refer¬ 
ence  [4]  for  a  survey  of  the  automated  feature  extraction  meth¬ 
ods  on  Digital  Photogrammetric  Systems). 

One  of  this  tools  is  the  semi-automatic  road  network  caption,  as 
described  in  the  reference  [7],  developed  jointly  by  SYSECA 
and  the  French  Geographic  Institute  (I.G.N.).  This  algorithm  is 
interactive:  the  user  sets  the  beginning  point  of  the  extraction, 
and  the  direction  in  which  he  wants  to  extract  the  road,  and  the 
algorithm  finds  the  other  points  of  the  roads.  When  the  algo¬ 
rithm  fails  to  find  the  next  point,  it  stops,  and  the  user  can  cor¬ 
rect  the  points  extracted,  or  help  the  algorithm  by  giving 
manually  some  points  (e.g.:  where  the  road  is  interrupted  by  a 
bridge).  The  points  extracted  are  given  in  a  3D  referential. 

For  the  building  extraction,  which  is  a  tedious  task  in  3D  site 
modelling,  the  algorithms  we  developed  are  based  on  a  Model- 
Based  Optimisation  paradigm,  and  allow  the  extraction  of  the 
so-called  extruded  objects  as  well  as  the  parametric  objects 
(parallelepipeds,  pyramids,  prisms,  cylinders,  ...).  These  algo¬ 
rithms  deal  with  the  optimisation  of  the  roofs  of  the  objects  on 
the  images.  For  the  extruded  objects,  a  snake-like  approach  is 
used,  similar  to  the  one  described  by  P.  Fua  in  [3].  For  the  para¬ 
metric  objects,  an  original  least-squares  based  parametric  opti¬ 
misation  scheme  is  implemented.  An  example  of  use  of  the 
semi-automatic  extruded  building  roof  extraction  is  given  on 
figure  6.  One  can  see  the  automatic  3D  refinement  of  the  initial 
model  jointly  on  both  images  of  the  stereoscopic  pair. 

Another  very  helpful  tool  is  the  "semi-automatic  ground  posi¬ 
tioning".  The  user  indicates  a  rough  3D  initial  and  the  fine  3D 
position  is  found  automatically  using  a  local  correlation  algo¬ 
rithm.  This  tool  has  proven  to  be  reliable  enough  to  take  in 
charge  for  the  user  the  measurment  of  the  ground  altitude  need- 
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ed  for  the  building  acquisition,  which  increases  in  a  significant 
way  his  productivity. 


left  image  before  optimization  right  image 


after  optimization 


Figure  6  :  a  building’s  roof  optimization 

In  addition,  the  acquisition  modes  of  planimetric  or  volumetric 
objects  are  able  to  use  projections  of  new  objects  on  part  of  an¬ 
other  object.  It  is  also  possible  to  build  volumes  by  copy-pasting 
objects,  which  is  useful  for  areas  where  several  buildings  are 
based  on  the  same  model  (e.g.  suburban  areas  with  peaked  roof 
houses). 

The  main  advantage  of  these  tools  is  the  automatic  estimation  of 
the  heights  of  the  points,  which  appears  to  be  a  tedious  task  for 
non-photogrammetrist  users.  It  can  also  be  seen  as  a  way  of  re¬ 
laxing  the  operator’s  attention  on  buildings  where  the  algorithm 
is  supposed  to  work  (good  contrast),  so  that  he  can  focus  on 
"difficult"  objects. 


3.  RENDERING 

This  stage  is  divided  in  two  steps:  the  Texture  Mapping  and  the 
Terrain  Integration. 

3.1.  Texture  mapping 

With  terrestrial  photos,  the  user  is  supposed  to  extract  by  hand 
the  textures  from  the  image,  e.g.  by  pointing  the  corners  for  a 
rectangular  face,  and  make  the  association  with  a  face  of  the  3D 
model.  Whereas  with  the  registered  aerial  or  satellite  images, 
which  have  been  used  as  a  support  for  the  geometric  reconstruc¬ 
tion,  this  process  can  be  automatic.  Such  an  approach  allows  the 
complete  texturation  of  the  site,  because  the  vertical  or  quasi¬ 
vertical  satellite  or  aerial  images  bring  the  terrain  and  roofs  tex¬ 
tures,  whereas  the  terrestrial  photos  give  the  textures  of  the 
walls  of  the  buildings. 

The  texture  mapping  process  is  highly  interactive  :  the  operator 
must  identify  on  the  terrestrial  photos  the  images  patch  corre¬ 
sponding  to  the  faces  of  the  3D  objects  extracted  in  the  geomet¬ 
ric  reconstruction  stage.  Such  a  human  identification  could  be 
avoided  if  the  photos  were  possible  to  register  as  the  aerial  or 
satellite  images.  The  main  obstacle  to  the  photogrammetric  ex¬ 
ploitation  of  such  photos  is  the  type  of  camera  generally  used  ; 
these  cameras  cannot  be  qualified  of  metric  cameras,  which 


means  their  geometric  model  cannot  be  known  and  estimated 
with  a  precision  compatible  with  the  required  reconstruction 
quality.  That’s  why  we  consider  the  texture  mapping  as  an  in¬ 
teractive  step. 

As  it  is  symbolized  on  figure  7,  this  step  consists  in  extracting 
the  image  patch  to  be  mapped,  eventually  retouching  this  patch, 
and  finally  mapping  this  texture  to  the  corresponding  face  of  the 
object.  The  retouching  step  can  be  useful  to  correct  the  artefacts 
caused  by  trees,  signs,  or  cars  on  face’s  texture,  and  to  clean  the 
edges  of  the  extracted  textures  ;  a  classical  software  product  like 
Photoshop  is  able  to  cope  with  these  problems  with  efficiency. 
It  has  to  be  noticed  that  the  texture  mapping  step  includes  the 
warping  of  the  image  patch  to  make  it  fit  in  the  geometry  of  the 
face  (a  rectangle  in  the  example  of  figure  7). 

For  faces  not  seen  on  photos,  it  is  possible  to  map  a  synthetic 
texture,  or  a  texture  previously  extracted  on  other  images. 
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Figure  7  :  Principles  of  texture  mapping 


3.2.  Terrain  integration 

Terrain  integration  represents  the  formatting  of  the  data  collect¬ 
ed  to  allow  a  realistic  representation  of  the  site.  It  uses  the  fol¬ 
lowing  steps: 

—  generic  objects  addition:  a  database  of  generic  objects  pro¬ 
vides  the  full  3D  description  of  the  objects  that  have  been  ex¬ 
tracted  as  bounding  boxes ;  the  resizing  of  the  3D 
representation  is  done  automatically  using  the  bounding  box; 

—  planimetry  formatting :  the  planimetric  objects  (roads,  ri¬ 
vers,  ...)  are  formatted  using  pre-defined  "gabarits"  (templa¬ 
tes  for  transversal  representation  of  lineic  primitives,  like 
roads  or  rivers),  insuring  a  realistic  representation; 

—  terrain  triangulation  :  the  DTM  must  be  formatted  to  be  vi¬ 
sualized;  this  formatting  consists  in  computing  a  triangular 
patches  set  that  represent  the  terrain  in  the  most  efficient 
manner;  this  triangulation  is  done  coherentely  with  the  ex¬ 
tracted  3D  objects,  which  insures  the  triangulated  terrain  is 
correctly  connected  to  roads  elements  and  buildings  bases 
(Constrained  Delaunay  Triangulation); 

—  orthophoto  mapping  :  the  orthophoto  is  automatically  map¬ 
ped  on  triangular  patches  of  the  terrain,  as  well  as  on  the 
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roofs  not  textured  yet;  in  order  to  prevent  from  the  lack  of 
realism  of  the  orthophoto  textures  at  short  distance  (due  to 
their  resolution),  we  use  a  process  of  micro-texturation  (high 
frequency  textures)  to  insure  a  good  realism  of  the  anima¬ 
tions  when  the  point  of  view  is  too  close  to  the  ground. 

The  whole  process  of  Terrain  Integration  is  done  with  the  TT&S 
software  product  called  ModTer. 

At  this  stage,  the  3D  site  database  can  directly  be  used  by  a  sim¬ 
ulation  software  in  order  to  provide  realistic  views  and  anima¬ 
tion  around  the  site. 

4.  EXPLOITATION 
4.1.  Scenario  preparation 

In  this  step,  the  operator  determines  the  scenario  he  wants  to 
play  around  the  site.  It  is  based  on  the  TT&S  software  Track- 
Maker.  The  user  can  define  the  mobile  object  which  is  supposed 
to  be  used  :  helicopter,  aircraft,  tank,  car,  or  even  a  pedestrian. 
Each  mobile  object  is  associated  to  a  specific  behaviour,  insur¬ 
ing  a  realisitic  animation.  The  user  then  defines  the  trajectory  of 
the  object,  as  well  as  the  climatic  conditions  of  the  animation 
(weather,  time). 


As  we  will  see  below,  the  definition  of  trajectories  is  not  indis¬ 
pensable  for  the  animation,  because  the  operator  can  decide  to 
move  in  real  time  around  the  site  using  the  keyboard  and  the 
mouse. 

4.2.  Animation 

The  animation  step  exploits  the  3D  site  database  to  simulate 
with  a  great  realism  the  scene  that  would  be  viewed  from  any 
point  around  the  site.  This  exploitation  is  based  on  the  TT&S 
software  SpaceMagic,  which  is  one  of  the  world-wide  referenc¬ 
es  in  the  domain  of  simulation,  offering  high  level  performances 
in  animation. 

This  software  exploits  the  geometric  models  and  the  textures,  to 
provide  a  totally  fluent  display  of  the  reconstmcted  scene  as  the 
user  moves  around  the  site.  The  scene  is  computed  and  re¬ 
freshed  in  real  time.  It  is  possible  to  use  a  predefined  scenario, 
or  the  user  can  move  interactively  in  real  time. 

This  animation  software  can  produce  fly-througli  animations, 
simulating  the  scene  viewed  by  a  flying  observer,  but  also  the 
so-called  walk-through  animations,  which  can  be  a  base  for  ter¬ 
restrial  mission  preparation  or  training. 

Note  that  the  ouptut  of  the  animation  can  be  stored  on  video¬ 
tape  or  CD-ROM  to  be  played  in  an  extern  context. 


Figure  8:  source  images 
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5.  EXAMPLE  OF  RECONSTRUCTION  The  figure  8  represents  a  split  view  of  the  stereoscopic  pair,  as 

well  as  one  of  the  helicopter  view  (which  is  actually  coloured). 

The  example  we  present  here  is  based  on  a  data  set  collected  on 
a  village  of  the  south-east  of  France.  This  data  comprises  : 

-  a  stereoscopic  pair  of  high  resolution  aerial  images  (typical-  °n  flSure  9’  one  can  see  the  result  of  the  geometric  reconstruc- 

ly  with  a  resolution  of  1  meter),  taken  with  a  metric  camera  tion  staSe-  The  reconstructed  site  can  be  seen  on  the  CIS  win- 

system  (scanned  black&white  photos);  dow  (which  is  a  cartographic  representation  of  the  site)  and  in 

the  perspective  window.  As  it  can  be  seen,  the  perspective  win- 
several  colour  images,  taken  from  a  non  metric  camera  fixed  dow  -s  abje  t0  display  the  3D  models  with  the  DTM  (represent- 

on  an  helicopter,  flying  at  low  altitude  (allowing  grazing  an-  e(j  wjt|1  wireframes)  or  with  the  orthophoto  drapped  on  the 

gles  of  view).  DTM.  This  latter  representation  gives  a  realistic  view  of  the  re- 

Such  a  combination  of  images  allow  the  full  demonstration  of  constructed  site,  but  does  not  include  the  textures  on  buildings, 

the  capabilities  of  our  system.  The  aerial  images  are  used  to  This  could  be  seen  as  the  result  of  the  geometric  reconstructing 

geometrically  reconstruct  the  site,  whereas  the  helicopter  views  stage, 
allows  the  extraction  and  mapping  of  real  textures  on  walls  and 
roofs. 


Figure  10:  result  of  the  3D  site  reconstruction  (geometry  and  textures) 
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Figure  11:  snaphots  of  a  terrestrial  mission  preparation  scenario 


The  final  model  site,  including  the  wall  textures  can  be  seen  on 
figure  10.  Note  that  this  view  is  presented  in  grey  level  in  this 
article  for  printing  constraints.  In  fact,  the  textures  that  have 
been  mapped  on  the  objects  and  the  terrain  are  coloured,  and 
give  a  realism  even  better  to  the  reconstructed  scene. 

For  the  experimentation,  we  built  a  terrestrial  mission  repetition 
scenario,  including  helicopters,  tanks  and  ground  troops.  Two 
snapshots  of  this  scenario  are  shown  on  figure  1.  On  one  of 
these  snapshots,  one  can  see  a  street  of  the  village  as  it  would  be 
seen  by  a  tank  at  ground  level.  The  other  snapshot  represents  a 
complex  scene  of  the  scenario,  where  tanks,  troups  and  helicop¬ 
ters  are  evoluating  on  the  site. 


CONCLUSION 

We  described  an  image-based  system  for  3D  site  model  acqui¬ 
sition  and  exploitation,  dedicated  to  the  production  and  exploi¬ 
tation  of  realistic  3D  site  databases. 

This  system  comprises  a  geometric  reconstruction  module,  al¬ 
lowing  the  extraction  of  3D  objects  and  terrain  from  a  set  of 
aerial  or  satellite  high  resolution  images,  and  a  rendering  step, 
where  textural  information  from  any  image  are  mapped  on  the 
extracted  objects.  These  two  steps  provide  3D  databases  that 
can  be  used  by  the  exploitation  module  of  the  system,  based  on 
a  powerful  simulation  software.  This  system  has  been  proto¬ 
typed,  and  the  results  have  demonstrated  the  capabilities  of  our 
approach. 

With  this  prototype,  we  show  that  an  autonomous  3D  databases 
production  line  from  images  is  feasible,  and  demonstrated  its  in¬ 
terest  from  an  operational  point  of  view.  The  availability  in  a 
near  future  of  more  and  more  sources  of  high  resolution  images 
will  certainly  emphasizes  the  need  for  such  systems. 
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DISCUSSOR’S  NAME:  M.  Desbois 

COMMENT/QUESTION: 

Utilisez-vous  des  moyens  de  designation  tri-dimensionels  pour  saisir  l’information  et  l’integrer 
dans  la  simulation? 

(Do  you  use  three-dimensional  designation  facilities  to  input  data  and  incorporate  it  into  the 
simulation?) 


AUTHOR/PRESENTER’S  REPLY: 

Deux  boules  roulantes  (x,y)  (z)  permettent  la  designation  3D. 


(Yes,  we  use  two  tracking  balls  (x,y)  and  (z)  for  3D  designation.) 
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1.  SUMMARY 

In  the  framework  of  a  NATO  measurement  campaign  air¬ 
borne  signature  measurements  were  conducted  over  sea 
with  the  dual  frequency,  polarimetric  mmW  Synthetic  Aper¬ 
ture  Radar  MEMPFIIS  (Millimeterwave  Experimental  Mul¬ 
tifrequency  Polarimetric  High  Resolution  Imaging  System) 
with  simultaneous  operating  front-ends  at  35  GHz  and  94 
GHz  onboard  a  cargo  aircraft  in  side-looking  configuration. 
Both  front-ends  are  tied  to  the  same  system  reference  and 
are  using  the  same  IF  pre-processing  and  radar  waveform- 
generator.  So  as  well  a  concise  comparison  between  data  at 
the  two  frequency  bands,  35  GHz  and  94  GHz,  can  be  made 
as  a  use  of  the  data  for  multichannel/multifrequency  SAR 
processing. 

The  paper  describes  the  system  configuration  and  the 
mmW-SAR  processing  algorithm  and  gives  representative 
results  for  the  generated  radar  images  for  ship  targets,  chaff 
and  the  sea  clutter  with  emphasis  on  the  multiparameter 
evaluation. 

2.  INTRODUCTION 

Fligh  resolution  mmW-radar  sensors  are  candidates  for 
autonomous  guidance  applications  for  missiles  directed 
against  maritime  targets.  In  a  close-in  phase  of  an  attack  it  is 
essential  for  such  seekers  to  discriminate  between  target  and 
chaff.  To  be  able  to  optimize  self  defence  strategies  as  well 
as  the  performance  of  chaff  against  such  sensors  it  is  neces¬ 
sary  to  know  the  highly  resolved  signature  of  naval  targets 
and  chaff  in  a  natural  environment.  In  the  final  phase  of  the 
approach  of  an  anti-ship  missile  seeker  also  steeper  depres¬ 
sion  angles  may  be  relevant.  For  this  geometry  the  contrast 
between  target,  chaffs/decoys  sea-clutter,  including  the 
impact  imposed  by  the  propagation  medium  (  multipath 
effects  )  have  to  be  known.  Only  experiments  with  airborne 
sensors  can  deliver  this  necessary  information. 


In  dependence  on  the  application,  the  systems  can  be  used 
with  polarimetric  monopulse  feeds,  sensing  elevation  and 
transverse  deviations  or  an  interferometric  pair  of  antennas 
with  orthomode  transducers  to  sense  both  polarimetric 
components.  The  elevation/azimuth-asymmetry  of  the 
beam,  which  is  generally  necessary  for  the  SAR  application 
is  achieved  by  an  aspheric  lens  in  front  of  the  feed  horns. 

A  more  detailed  description  of  the  radar  was  already  given 
elsewhere  [1].  MEMPHIS  is,  to  our  knowledge,  the  only 
SAR-system  capable  to  gather  full  polarimetric  data  at  both 
relevant  mmW  bands  simultaneously. 

The  performance  data  of  the  front-ends  are  summarised  in 
Tab.  3.1: 


3.  Measurement  Set-Up 

3.1  Experimental  Radar 

As  mentioned  above,  the  SAR  system  employs  two  radar 
front-ends,  one  at  35  GHz  and  the  other  at  94  GHz.  Both  are 
controlled  by  a  common  VME-Bus  computer  and  tied  to 
the  system  reference,  from  which  all  frequencies  and  trigger 
impulses  used  in  the  system  are  derived.  Each  receiver  has 
four  channels.  The  downconverted  signals  are  quadrature 
demodulated  to  result  in  I-  and  Q-phase  components  and  the 
logarithmically  weighted  amplitudes.  The  IF-signals  from 
both  front-ends  are  fed  to  the  data  acquisition  and  recording 
electronic.  Additional  information  is  retrieved  from  the 
inertial  navigation  system  of  the  aircraft  and  a  separate 
GPS-receiver  unit.  Fig.  3.1  shows  a  block  diagram  of  the 
total  set-up. 


Fig.  3.1:  Block-Diagram  of  the  35/94-GHz-SAR 
"MEMPHIS" 


The  measured  data  are  recorded  by  means  of  a  high  speed 
digital  recording  machine  AMPEX  DCRSi.  The  recording 
speed  is  limited  to  33  MByte/s  allowing  to  record  660  range 
gates  at  a  PRF  of  6  kHz  for  both  transmit  frequencies. 

In  parallel  to  the  radar  data  inertial  data  from  the  aircraft  as 
well  as  a  time  code  and  GPS-data  are  recorded. 

The  calibration  is  based  on  pre-  and  postflight  measure¬ 
ments  against  trihedral  and  dihedral  precision  corner  reflec¬ 
tors. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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1  Transmitter: 

Centre  Frequency  94  GHz  35  GHz 

Output  Power 

1  kW  900  W 

PRF 

6  kHz  ( typ.) 

Pulse  Width 

80  ns  -  2  ps 

Spectral  Purity 

>  -  70  dB  /  Hz 

Phase  Stability 

>  10°  RMS 

Polarization 

switchable  from  Pulse  to  Pulse 

H-V  or  LC-RC 

switchable  manually  linear  or 
circular 

Waveform 

Receiver: 

Polyphase  Franck  Code 

Four  identical  channels  for  co-  and  cross-polarisation, 

Monopulse  or 

Interferometry 

Dynamic  Range 

60  dB 

System  Noise  Figure  15  dB  (SSB) 

Antenna: 

Type  Monopuls  or  Interferometric  Pair 

Dielectric  tense 
Diameter 

30  cm 

3-dB-Beam  Width 

Azimuth 

1°  (94  GHz)  2.5°  (35  GHz) 

Elevation 

12°  16° 

Gain 

29  dB  27  dB 

Data  Acquisition: 

Data  recording  : 

AMPEX  DCRSi 

Data  Rate 

33  MByte  /  sec 

Channels 

8  +  flight  parameters, 
time-code,  GPS,  etc. 

Capacity 

48  G  Byte 

Tab.  3.1:  Performance  Data  of  the  Experimental  Radar 
MEMPHIS 

3.2  Geometry 

The  measurements  discussed  below  were  conducted  in  side¬ 
looking  geometry. 

Typical  geometrical  and  flight  parameters  are  tabulated  in 
Tab.  3.2 . 


Speed  of  Aircraft 

130  kn. 

Height  above  Ground 

1000  ft 

Depression  Angle 

20° 

Slant  Range  at  Beam  Centre 

770  m 

Swathwidth 

406  m 

Tab.  3.2  :  Typical  Measurement  Parameters  for  mmW- 
SAR-Imaging 

To  be  able  to  give  the  signature  of  ship  for  the  main  aspect 
angles  of  the  ship  -  backboard  and  starboard,  bow  and  stern 
-  the  aircraft  flew  along  a  track  as  indicated  in  Fig.  3. .2. 

The  aircraft  flew  with  a  lateral  displacement  of  770  m  from 
the  target,  while  the  ship  moved  slowly  with  a  constant 
speed  and  held  a  steady  and  straight  course  throughout  a 
trial  run.  To  manoeuvre  the  aircraft  into  a  stable  straight 
course  the  aircraft  had  to  fly  loops  as  indicated  in  the  sketch. 
One  total  measurement  run  on  a  ship  took  about  4  minutes. 


Target  tracking  was  achieved  manually  using  a  boresight 
video  image  of  the  target,  within  the  radar’s  range  gate. 

Also  for  the  characterization  of  chaff-clouds  a  straight  flight 
path  with  a  correct  lateral  displacement  was  necessary.  Due 
to  the  limited  swath-width  of  only  about 
400  m,  the  correct  position  of  the  chaff  and  the  wind  direc¬ 
tion  was  important  to  adjust  the  lateral  displacement  of  the 
aircraft  track. 


Fig.  3.2:  Sketch  of  Flight  Path  during  the  Measurements 

4.  EVALUATION  METHODS 

The  evaluation  employed  image  based  methods,  namely  a 
synthetic  aperture  algorithm.  Based  upon  those  methods, 
images  areas  of  interest,  containing  clutter  or  targets  were 
selected  to  apply  integral  evaluation  methods.  In  addition  to 
this  approach  circular  flight  patterns  as  suggested  by  Jones 
at  Langley  [2]  were  used  to  generate  radar  data  in  a  reflec- 
tometer  type  of  radar  mode  with  the  aim  to  gather  clutter 
data  for  the  full  aspect  angle  range  of  360°  with  respect  to 
the  direction  of  the  sea  swell. 

4.  1  Image  Generation 

In  a  first  step  the  radar  data  undergo  off-line  track,  calibra¬ 
tion,  and  reformatting  procedures.  Afterwards  synthetic 
aperture  processing  is  applied  to  these  data  to  generate  radar 
images  for  co-  and  cross-polarisation  and  both  at  35  GHz 
and  94  GHz. 

As  already  discussed  elsewhere  [3]  SAR-processing  at 
millimeterwavelengths  requires  a  considerable  lower 
amount  of  sophistication  in  comparison  with  algorithms 
applied  at  lower  radar-frequencies.  This  can  mainly  be 
attributed  to  the  short  aperture  length  at  mm-wave  fre¬ 
quencies.  Taking  this  into  account,  the  SAR-algorithm  used 
here  is  relatively  simple  although  fully  automatic  autofo¬ 
cussing  is  applied,  using  only  radar-data  without  supply  of 
external  inertial  navigation  information. 

4.2  Integral  Evaluation 

The  integral  evaluation  covered  the  determination  of  sta¬ 
tistical  characteristics  of  ships  and  clutter,  and  its  depend¬ 
ence  on  the  radar  band,  35  GHz  and  94  GHz,  and  also  on 
aspect  angle  and  environmental  conditions.  Special  care 
was  taken  on  polarimetric  features,  which  are  well  recog¬ 
nised  means  to  enhance  image  contrast  [4]  and  to  do  classi¬ 
fication  [5]  according  to  the  scattering  mechanisms  in¬ 
volved. 
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5.  Results 

5.1  Sea  Clutter 

Figs.  5.1  to  5.3  show  plots  for  the  reflectivity  of  the  sea 
surface  versus  aspect  angle  with  respect  to  the  wave  direc¬ 
tion  for  the  radar  frequencies  35  GHz  and  94  GHz  and 
polarization  states  VV  and  HH,  VH  and  VV.  The  curves 
apply  to  sea  state  5  and  open  sea  conditions. 


Fig.  5.1:  Reflectivity  versus  Aspect  Angle 

Radar  Frequency  35  GHz,  T/R-Polarizations  VV  and  HH. 


Fig.  5.2:  Reflectivity  versus  Aspect  Angle  Radar  Frequency 
94  GHz,  T/R-Polarization  VV 


Fig.  5.3:  Reflectivity  versus  Aspect  Angle 

Radar  Frequency  35  GHz,  T/R-Polarizations  HV  and  VH. 

The  figures  show  the  periodic  pattern  of  growing  and  de¬ 
creasing  reflectivity  dependent  on  the  wave  aspect,  i.e.  up¬ 
wind,  down-wind  or  cross-wind. 

The  experimental  curves  were  modelled  by  the  function  [6] 
cr0  =  A  +  B  cos  4>  +  C  cos  2<J)  [m2/m2], 
where  <j>  is  the  horizontal  angle  between  the  radar  look 
direction  and  the  upwind  direction.  The  coefficients  A,  B 
and  C  are  functions  of  the  depression  angle  0,  wind  speed  u, 
radar  wavelength  X  and  polarization  p  and  can  be  expressed 
by  means  of  a  factor  times  the  windspeed  with  the  wind 
speed  factor  y: 

A  =  a(X,6,<t>,p)  uTa(A"9'lt>’p) 

B  =  b(X.,0,<t>,p)  urb(W'p) 

C  =  c(^,0,<ji,p)  uyc(X'0Ap) 

The  model  curves  were  fitted  to  the  experimental  curves  and 
are  shown  in  the  figures  together  with  the  experimental  data 
using  the  relations  between  o0  at  up-wind  (u),  down-wind 
(d)  and  cross-wind  (c)  which  are  easily  derived  : 

G°u+a°  j+2g°c 
A  = - 

4 

B  = - - 

2 

G°u  +o°  d  -2a°c 

c  = - 

4 

The  parameters  were  extracted  for  the  available  polarization 
states  and  radar  frequencies  and  a  depression  angle  of  37.5°. 
They  are  tabulated  in  Tab.  5. 1 . 

no  at  94  GHz  reproduces  very  well  the  number  ,  which  was 
measured  for  similar  sea  conditions  during  earlier  measure¬ 
ments  [7]  for  which  the  wind  speed  exponent  was  derived  to 
be  y  =  2.3. 
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T/R- 

Polarizati 

on 

H-H 

H-V 

V-V 

V-H 

35  GHz 

A 

0.0038 

0.4125E-3 

0.005 

0.4125E-3 

B 

0.0038 

0.1  E-3 

0.0025 

0.125  E-3 

C 

0.0025 

0.1375  E- 
3 

0.0035 

0.1625  E- 
3 

94  GHz 

A 

0.0068 

B 

0.0035 

C 

0.0028 

Tab.  5.1:  Model  Factors  for  ao  versus  Aspect  Characteristic 
at  Sea  State  5 


To  our  knowledge  these  measurements  are  the  only  full 
polarimetric  millimeterwave  measurements  of  this  kind, 
which  allow  to  assess  the  frequency  dependence  of  the 
model  factors.  If  compared  with  respective  X-band  results 
[8]  the  principle  characteristics,  especially  at  35  GIIz,  are 
very  similar.  At  94  GHz  different  dominant  scattering  proc¬ 
esses  lead  to  a  significant  change  especially  for  the  coeffi¬ 
cient  B,  which  is  related  to  the  up-wind/down-wind  ratio. 
General  results  were  the  following: 

•  sharp  cross  wind  nulls  are  appearing 

•  the  reflectivity  at  down-wind  is  always  lower  than  at  up¬ 
wind.  The  up-wind/down-wind  ratio  is  dependent  on  the 
parameters  X,0,(j>,p. 

•  the  co-polar  signal  at  vertical  polarization  is  stronger 
than  the  co-polar  return  at  horizontal  polarization.  The 
ratio  can  range  up  to  10  dB.  This  different  characteristic 
is  more  pronounced  for  low  sea  states  than  for  rough 
sea. 

•  at  94  GHz  the  reflectivity  at  down-wind  is  close  to  that 
at  cross  wind. 

All  these  results  can  be  explained  by  a  scattering  model  for 
the  interaction  between  the  sea  surface  and  mm-waves, 
which  is  based  on  a  scattering  dominated  by  small  facets 
located  in  the  crests  of  the  waves  where  the  rise  of  the  sur¬ 
face  is  steep  and  multiple  scattering  between  facets  is  fre¬ 
quent. 

5.2  Ships  and  Chaff 

5.2.1  SAR-Imaging  of  Ships  and  Chaff 

Measurements  of  ships  and  ships  firing  chaff  have  been 
done  at  35  GHz  and  at  94  GHz  with  linear  polarisation 
and  with  a  fixed  radar  depression  angle  of  22.5°. 

Fig.  5.4  and  5.5  show  SAR  images  at  35  GHz  and  94  GHz 
for  a  typical  scene  with  bloomed  chaff  clouds  and  the  cor¬ 
responding  ship. 

The  SAR  images  indicate  where  dominant  scatterers  on  the 
ship  are  located.  There  exist  cases,  where  due  to  multipath 
effects  signal  portions  are  reflected  from  the  ship's  body 
(dominant  scatterers  on  the  ship)  into  the  water  surface 
reproducing  reflected  radar  images  along  the  water  line  of 
the  ship.  This,  however  could  be  observed  only  for  35  GHz 
at  calm  sea  conditions. 


Fig.  5.4  SAR  Image  of  Chaff  and  Ship  at  35  GHz 


Fig.  5.5:  As  Fig.  5.4  at  94  GHz 

Generally  the  measurements  showed  that  the  scattering 
centers  appear  more  concentrated  at  94  GHz,  while  at  35 
GHz  the  scatterers  produce  wider  lobes. 

The  signatures  for  chaff  are  considerably  less  wide  spread  at 
94  GHz  if  compared  with  35  GHz.  It  has,  however  to  be 
noted,  that  the  types  of  chaff  measured  here  were  not  spe¬ 
cially  designed  against  mm-wave  sensors. 

5.2.2  Statistical  evaluation  of  Ship -  and  Chaff-Data 
Radar  cross  section  evaluations  have  been  carried  out  on 
different  ship  classes  and  different  displacements  of  the 
ships  by  extracting  statistical  data  from  the  SAR  images.. 
The  experimental  results  of  radar  cross  sections  at  35  GHz 
and  94  GHz  are  plotted  in  Fig.  5.6  versus  ship  type  . 

It  can  be  seen  that  the  measured  average  radar  cross  section 
of  ships  depends  on  the  type  of  ship  and  also  on  the  radar 
frequency. 

For  comparison  reasons  Fig.  5.7  presents  the  average  RCS 
of  ships  versus  the  displacement,  given  in  tons. 

Usually  the  RCS  of  ships  is  expected  to  increase  with  fre¬ 
quency  [8],  thus  at  Ka-band  the  RCS  should  be  lower  than 
at  W-band.  However,  the  expected  increase  of  RCS  with 
frequency  was  not  observed  as  in  earlier  ground  based 
measurements  [9]. 
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94  GHz,  respectively.  At  35  GHz  the  average  RCS  of 
chaffs(  27  dBsm  to  30  dBsm)  was  about  5  dB  higher  than 
the  RCS  of  chaff-clouds  at  94  GHz. 

In  general  the  RCS  of  ship  (  broadside  aspect  )  compared 
with  the  RCS  of  chaff-clouds  was  about  6  dB  higher  at  both 
frequencies.  For  signal  lock-transfer,  the  RCS  of  the  chaff- 
clouds  should  be  preferably  higher  than  the  ship’s  RCS. 
Therefore,  lock-  transfer  with  chaff  as  used  here  at  millime¬ 
ter  waves  would  hardly  be  feasible. 


a  )  35  GHz 


Run  of  Ship  and  Type  of  Chaff 


Fig.  5.6:  Average  RCS  of  different  Ships  of  same  Class  at 
35  GHz  (a)  and  94  GHz  (b) 


Fig.  5.7:  Average  RCS  versus  Displacement 

In  contrary,  comparing  both  frequency  bands,  the  radar 
cross  section  at  35  GHz  is  about  6  dB  higher  than  the  RCS 
at  94  GHz  for  almost  all  classes  of  ships  considered  here. 
This  can  be  interpreted  by  the  following:  Compared  with  the 
radar  wavelength  (  X  =  3  mm  and  X  =  8.6  mm  )  flat  plates 
and  corners  on  a  ship’s  body  are  not  smooth  and  straight, 
therefore,  they  can  not  be  looked  at  as  basic  dominant  scat- 
terers  with  ideal  shape.  Moreover  at  millimeter  wavelength 
not  well  defined  scatterers  determine  the  signature  but  the 
whole  community  of  numerous  point  source  scatterers  are 
considered  to  be  the  coherent  average  sum  of  their  individ¬ 
ual  parts  [8]. 

Fig.  5.8  presents  the  measured  RCS  of  chaffs  at  35  and  94 
GHz  at  linear  polarisation:  HH.  The  highest  cross  section 
was  measured  to  be  about  35  dBsm  at  35  GHz  and  at 
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Fig.  5.8:  Average  RCS  of  different  Types  of  Chaff  for  35 
GHz  (a)  and  94  GHz  (b) 

6.  CONCLUSIONS 

In  the  mmW  radar  bands  (Ka-band  and  W-band  )  Synthetic 
Aperture  Radar  (SAR)  systems  offer  attractive  features  for 
remote  sensing  applications. 

The  system  used  here  is  capable  to  do  clutter  measurements 
over  sea  and  to  supply  the  reflectivity  data  needed  for  the 
development  of  maritime  mm-wave  sensors,  which,  al¬ 
though  typically  not  long  range,  will  gain  increasing  impor¬ 
tance  in  the  future.  The  discussion  of  typical  clutter  data 
showed  that  the  characteristics  of  35-GHz  sea  -clutter  seems 
to  be  very  similar  to  that  of  classical  radar  bands  and  that  at 
94  GHz  other  scattering  processes  are  dominant,  which  lead 
to  a  considerably  reduced  reflectivity  of  the  sea  surface  in 
down-wind  and  cross-wind  directions  and  an  increase  in  up¬ 
wind  direction. 

SAR  signature  measurements  on  ships  and  chaff  confirmed, 
what  was  already  known  from  landbased  measurements: 
The  RCS  of  ships  and  chaffs  depends  on  frequency,  on  the 
target  geometry,  and  the  type  of  chaff.  It  is  especially  re¬ 
markable  that  the  RCS  at  94  GHz,  in  contrary  to  theoretical 
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predictions,  is  lower  than  for  35  GHz  as  well  for  ships  as  for 
the  chaff. 

At  millimeter  wavelengths  not  distinct  scatterers  determine 
the  signature  of  a  target,  but  the  mutual  interaction  of 
numerous  scatterers  of  similar  importance  are  dominantly 
contributing  to  the  total  radar  cross  section. 

For  self  protection  of  a  ship,  RCS  reduction  of  the  ship  will 
favour  lock-transfer  to  chaff. 

The  images  showed,  that  SAR  at  mmW  exhibits  certain 
characteristic  features: 

-  due  to  the  relatively  rough  surfaces  on  a  ship,  compared  to 
the  radar  wavelength,  dominant  structures  or  scatterers' 
exhibit  no  excessive  backscatter  cross-sections. 

-  the  total  radar  cross  section  of  ships  is  about  6  dB  higher 
than  the  RCS  on  chaff  clouds 
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PAPER  No:  5 

DISCUSSOR’S  NAME:  Streng 

COMMENT/QUESTION: 

1 .  How  did  you  handle  the  occurrence  of  ghost-scatterers?  (In  order  to  be  able  to 
automatically  classify  ships?) 

2.  Did  you  implement  this? 

AUTHOR/PRESENTER’S  REPLY: 

1 .  Compare  with  ships’  model,  several  imges  under  various  aspect  angles. 


2.  May  be  done  in  the  future. 
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SUMMARY 

An  approach  to  the  long  range  automatic  detection  of  vehicles, 
using  multi-sensor  image  sequences  is  presented.  The  algorithm 
we  use  was  tested  on  a  database  of  six  sequences,  acquired 
under  diverse  operational  conditions.  The  vehicles  in  the 
sequences  can  be  either  moving  or  stationary.  The  sensors  are 
stationary,  but  can  perform  a  pan/tilt  operation.  The  presented 
paradigm  uses  data  fusion  methods  at  four  different  levels 
(feature  level,  sensor  level,  temporal  level  and  decision  level) 
and  consists  of  two  parts. 

The  first  part  detects  targets  in  individual  images  using  a  semi- 
supervised  approach.  For  each  type  of  sensor  a  training  image  is 
chosen.  On  this  training  image  the  target  position  is  indicated. 
Textural  features  are  calculated  at  each  pixel  of  this  image. 
Feature  level  fusion  is  used  to  combine  the  different  features  in 
order  to  find  an  optimal  discrimination  between  target  and  non¬ 
target  pixels  for  this  training  image.  Because  the  features  are 
closely  linked  to  the  physical  properties  of  the  sensors,  the  same 
combination  of  features  also  gives  good  results  on  the  test 
images,  which  are  formed  of  the  remainder  of  the  database 
sequences.  By  applying  feature  level  fusion,  a  new  image  is 
created  in  which  the  local  maxima  correspond  to  probable  target 
positions.  These  images  coming  from  the  different  sensors  are 
then  combined  in  a  multi-sensor  image  using  sensor  fusion.  The 
local  maxima  in  this  multi-sensor  image  are  detected  using 
morphological  operators.  Any  available  prior  knowledge  about 
possible  target  size  and  aspect  ratio  is  incorporated  using  a 
region  growing  procedure  around  the  local  maxima.  A  variation 
to  this  approach,  that  will  also  be  developed  in  this  paper, 
combines  the  previous  feature  and  sensor  level  fusion,  by 
extracting  the  features  in  each  sensor  as  before  but  using  the 
feature  level  fusion  directly  on  the  combination  of  all  features 
from  all  sensors  in  what  is  sometimes  called  a  « super  feature 
vector ».  Tracking  is  used  in  both  cases  to  reduce  the  false  alarm 
rate. 

The  second  part  of  the  algorithm  detects  moving  targets.  First 
any  motion  of  the  sensor  itself  needs  to  be  detected.  This 
detection  is  based  on  a  comparison  between  the  spatial  co¬ 
occurrence  matrix  within  one  single  image  and  the  temporal  co¬ 
occurrence  matrix  between  successive  images  in  a  sequence.  If 
sensor  motion  is  detected  it  is  estimated  using  a  correlation- 
based  technique.  This  motion  estimate  is  used  to  warp  past 
images  onto  the  current  one.  Temporal  fusion  is  used  to  detect 
moving  targets  in  the  new  sub-sequence  of  warped  images. 
Temporal  and  spatial  consistency  are  used  to  reduce  the  false 
alarm  rate. 

For  each  sensor,  the  two  parts  of  the  algorithm  each  behave  as 
an  expert,  indicating  the  possible  presence  of  a  target.  The  final 
result  is  obtained  by  using  decision  fusion  methods  in  order  to 


combine  the  decisions  of  the  different  experts.  Several  «k  out  of 
n  »  decision  fusion  methods  are  compared  and  the  results 
evaluated  on  the  basis  of  the  6  multi-sensor  sequences. 

1  INTRODUCTION 

Long  range  automatic  detection  of  vehicles  is  of  great  military 
importance  to  modem  armed  forces.  The  most  critical  factor  of 
any  system  for  automatic  detection  is  its  ability  to  find  an 
acceptable  compromise  between  the  probability  of  detection  (= 

1  -  probability  of  a  miss)  and  the  number  of  false  alarms.  This  is 
the  classical  trade-off  one  finds  in  binary  hypothesis  testing 
between  the  two  types  of  error  one  can  make :  the  false  rejection 
(FR  :  which  corresponds  here  to  a  miss :  there  is  a  target,  but  it 
has  not  been  found)  and  the  false  acceptance  (FA :  which  is  in 
this  case  the  same  as  a  false  alarm  :  there  is  no  target,  but  the 
system  thinks  there  is  one).  In  a  single  sensor  detection  system  it 
is  well  known  that  if  one  reduces  one  type  of  error,  the  other 
type  of  error  automatically  increases.  A  possible  way-out  of  this 
deadlock  is  to  use  more  than  one  sensor  and  to  combine  the 
information  coming  from  these  different  «  experts  ».  This 
combination  or  (data)  fusion  can  be  done  on  different  levels.  In 
this  paper,  only  the  (common)  case  of  a  centralised  fusion 
processor  with  all  its  sensors  connected  in  parallel  will  be 
considered. 

In  the  specific  data  fusion  literature  [1-5]  one  often  distinguishes 
between  the  following  (or  equivalent)  fusion  levels :  low  level 
fusion  (also  called  score  or  measurement  level  fusion),  medium 
level  fusion  (which  includes  feature  and  sensor  level  fusion), 
high  level  fusion  (also  called  decision  level  fusion)  and  temporal 
level  fusion.  As  can  be  expected,  in  real  (-time)  applications, 
there  is  a  trade-off  to  be  made  between  the  amount  of 
information  that  can  be  combined  and  the  bandwidth  necessary 
to  communicate  all  this  information  to  the  centralised  fusion 
processor.  The  lower  the  level  of  fusion,  the  more  information  is 
available  to  be  combined,  but  the  larger  becomes  the  bandwidth 
necessary  to  communicate  with  the  centralised  fusion  processor 
(or  for  a  fixed  bandwidth,  the  slower  becomes  the  fusion 
process).  Vice  versa  one  sees  that  when  the  level  of  fusion  gets 
higher,  the  available  information  diminishes,  but  so  does  the 
necessary  bandwidth.  Furthermore  not  all  data  fusion  levels  are 
always  applicable.  For  instance,  if  low  level  fusion  is  going  to 
be  used,  care  must  be  taken  to  combine  only  similar  entities 
(scores,  measurement  results,...).  It  is  therefore  impossible  to 
use  low  level  fusion  to  combine  the  raw  results  coming  from 
two  (or  more)  totally  different  sensors  (e.g.  an  imaging  sensor 
and  a  range  finder).  But  this  constraint  doesn’t  exist  any  longer 
on  the  decision  level,  where  each  sensor  is  considered  as  a 
separate  «  expert »,  who  decides  on  his  own.  In  the  special  case 
of  target  detection  where  the  «  hard  »  binary  decision  rule  is 
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used  (the  «  hard  »  decision  is  indeed  bi- valued  :  target  present 
(1)  or  not  (0)),  the  central  fusion  processor  contents  itself  to 
combine  only  the  decisions  (the  l’s  and  the  0’s)  coming  from 
different  sensors,  without  considering  the  type  of  sensor.  As  a 
general  conclusion  concerning  the  different  data  fusion  levels, 
one  can  state  that  all  different  fusion  levels  have  their 
importance  and  their  specific  applicability  domain. 

Based  on  these  considerations,  we  have  tried  to  use  data  fusion 
on  several  levels  to  try  to  optimise  the  use  of  the  available  data. 
That  is  basically  why  this  paper  describes  an  approach  to  tackle 
the  previously  exposed  problem  using  four  different  data  fusion 
techniques  related  to  several  levels :  feature  level  fusion,  sensor 
level  fusion,  temporal  level  fusion  and  decision  level  fusion. 

The  only  fusion  level  that  is  not  used  in  this  paper  is  the  low 
level  fusion.  This  technique  (in  the  form  of  pixel  level  fusion)  is 
mainly  used  in  remote  sensing  applications  [6,  7],  In  the  main 
approach,  we  do  however  use  two  different  medium  level  fusion 
techniques.  In  the  following  sections  the  use  of  these  different 
data  fusion  techniques  will  be  explained  in  more  detail. 

2.  IMAGE  DATABASE 


For  the  development  and  testing  of  the  algorithm,  a  database  of 
6  multi-spectral  image  sequences,  numbered  MS01  to  MS06 1 
was  compiled.  The  sequences  correspond  to  two  scenarios.  The 
first  scenario  is  a  typical  surveillance  scenario  in  which  the 
sensor  watches  a  scene  and  tries  to  detect  targets  entering  its 
field  of  view.  In  this  scenario,  the  targets  are  moving.  The 
second  scenario  is  a  reconnaissance  scenario  in  which  the  sensor 
is  mounted  in  a  new  terrain  and  it  tries  to  detect  the  presence  of 
vehicles  which  can  now  be  stationary  or  moving.  During  image 
acquisition  the  sensor  is  stationary  in  both  scenarios;  it  can  only 
perform  a  pan  and  tilt  operation.  The  following  table  presents 


some  properties  of  the  sequences. 


Seq 

nr 

Targets 

Target 

Motion 

Sc 

Type  of 
Sensors 

MS01 

Helicopter 

Tmck 

Across  FOV 

1 

LW,TV 

MS02 

Truck 

Towards 

Sensor 

1 

LW,TV 

MS03 

Across  FOV 

1 

LW,TV 

MS04 

2  Tanks 

None 

2 

LW,SW,TV 

MS05 

2  Tanks 
+Camoufl. 

None 

2 

LW,SW,TV 

MS06 

Across  FOV 

1 

LW,R,G,B 

3.  OVERVIEW  OF  THE  APPROACH 

The  proposed  algorithm  consists  of  two  independent  parts.  The 
first  part  searches  for  targets  in  single  images  while  the  second 
part  uses  multiple  subsequent  images  in  order  to  specifically 
find  moving  targets. 


Figure  1:  Global  Overview  of  the  method 


For  the  first  part  of  the  algorithm  we  have  chosen  a  semi- 
supervised  approach  based  on  texture  feature  extraction. 
Allthough  we  are  not  interested  in  explicitly  modelling  or 
measuring  texture,  these  texture  features  are  interesting  because 
they  are  independent  measurements  of  the  local  spatial 
distribution  of  grey  values  within  an  image  and  it  is  likely  that 
some  of  these  parameters  will  highlight  the  difference  between 
targets  and  background.  The  texture  parameters  are  even  more 
appealing  because  it  can  easily  be  seen  that  features  that  arc 
classically  used  for  target  detection  such  as  intensity  and 
gradient  are  just  special  cases  of  these  texture  parameters. 
Feature  level  fusion  is  used  to  combine  the  texture  features  from 
each  image  into  a  new  image  in  which  the  grey  value  at  each 
pixel  is  proportional  to  the  probability  that  the  pixel  belongs  to 
the  target.  These  images  from  the  different  sensors  are  fused  in  a 
sensor  fusion  step. 

The  second  part  of  the  algorithm  detects  moving  targets  in  sub¬ 
sequences  from  each  sensor  separately. 

Each  part  of  the  algorithm  behaves  as  an  expert  indicating  the 
possible  presence  of  vehicles  in  the  scene.  Decision  fusion  is 
used  to  combine  the  outcomes  from  all  experts. 


Table  1:  Properties  of  sequences. 

In  the  table  the  Sc  column  presents  the  scenario  to  which  the 
sequence  corresponds.  In  the  sensor  column  the  following 
abbreviations  are  used:  LW  and  SW  denote  long-wave  and 
short-wave  infrared  respectively.  TV  is  B/W  visual  images. 
R,G,B  are  the  components  of  a  colour  visual  image.  Each  set  of 
three  subsequent  sequences  were  acquired  by  the  same  sensor 
set.  Sequence  MS05  is  the  same  as  sequence  MS04  except  for 
the  fact  that  in  MS05  the  targets  are  camouflaged. 

1  MS01-MS03:  Courtesy  of  Defense  Research  Establishment 
Valcartier,  Quebec,  Canada  ;  MS04-MS05:  Courtesy  of  Naval 
Air  Warfare  Center,  China  Lake,  US;  MS06:  Courtesy  of 
ASIAT-DTT,  Peutie,  Belgium 


4.  TARGET  DETECTION  IN  SINGLE  IMAGES  fTPST 
Module ) 

Introduction 

For  the  detection  of  targets  in  single  images,  a  semi-supervised 
approach  based  on  texture  features  was  chosen.  For  each  sensor 
type,  one  image  was  selected  to  constitute  the  learning  database. 
On  these  images  the  true  targets  were  delimited.  Then  several 
texture  parameters  were  calculated  at  each  pixel  of  these 
learning  images  and  logistic  regression  [4]  was  used  to  find  a 
combination  of  the  texture  parameters  that  is  proportional  to  the 
probability  of  finding  a  target  at  the  corresponding  image 
location. 
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The  actual  detection  algorithm  then  applies  the  same 
combination  to  the  texture  features  calculated  on  the  remainder 
of  the  image  database  (test  images).  When  this  function  is 
applied  to  the  texture  features  calculated  at  each  pixel  of  a  test 
image,  a  new  image,  called  feature-level-fused  image,  can  be 
formed  in  which  the  maxima  correspond  to  likely  target 
positions.  These  feature-level-fused  images,  obtained  from  all 
the  different  sensors,  are  then  fused  again  in  a  subsequent  sensor 
fusion  step. 

To  find  the  possible  target  positions,  first  the  local  maxima  are 
determined  in  this  sensor-fused  image  and  then  available  prior 
knowledge  about  possible  target  size  and  aspect  ratio  is  used  to 
reject  false  targets. 


Figure  2:  TDSI  Module 
Texture  Parameters 


The  calculation  of  the  texture  features  is  based  on  the  co¬ 
occurrence  matrix.  The  co-occurrence  matrix  is  defined  as  a 
function  of  a  given  direction  and  distance,  or  alternatively,  as  a 
function  of  a  displacement  (dx,dy)  along  the  x  and  y  direction  in 
the  image.  For  a  given  displacement  (dx,dy),  the  (i,j)  element  of 
the  co-occurrence  matrix  is  the  number  of  times  the  grey  value 
G  at  the  current  position  (x,y)  is  i  when  the  value  at  the  distant 
position  (x+dx,y+dy)  is  j. 

=  P(G(x,y)=i  | G(x+dx,y  +  dy)  =  j) 

The  co-occurrence  matrix  can  be  calculated  on  the  whole  image. 
However,  by  calculating  it  in  a  small  window  scanning  the 
image,  a  co-occurrence  matrix  can  be  associated  with  each 
image  position.  The  centre  of  the  window  is  denoted  (xc,  y c)  and 
the  corresponding  co-occurrence  matrix  is  (i,j) 


Concurrence  Matrix  (for  dx=l,  dy  =2) 
Grey  Values  i 
0  1  2  3  4  ....256 

0  0  0  4  1  0  ...  0 

10  0  13 

2  4  10  3 

Grey  3  13  3  0 

Values  4  0  0  0  0  0  ...  0 

j 


256 1  0  0  0  0  0  ...  0 


Figure  3:  Co-occurrence  matrix 

In  Figure  3  an  example  of  a  co-occurrence  matrix  is  shown.  The 
matrix  corresponds  to  the  small  window  of  the  image  on  the  left 
and  was  calculated  for  a  displacement  of  dx  =  1 ,  dy  =  2.  The 
textural  features  that  were  used,  were  introduced  by  Haralick  [8- 
10]  and  are  widely  used  in  texture  analysis.  Based  on  the  local 
co-occurrence  matrix,  the  used  parameters  are  defined  as 
follows: 

F\  ( xc,yc )  =  Energy  =  ('J)2 

i  j 

F2 (xc,yc)  =  Contrast  =  XX[(;' ~  j)Ctfc 
<  J 

F3(xc,yc)  =  Max.Prob.=  max|c*^(i,y)j 
F4(xc,yc)  =  Entropy  =  (^')l0  g[Cwf  (i'j) 


>  j 


F5(xc,yc)  =  Homogeneity  = 

i  j 


max 


F6(xc,yc)  =  Variance  = 


with  Ej  =  X and 


We  are  not  interested  in  modelling  or  measuring  texture  but 
only  in  detecting  a  difference  between  target  and  background 
pixels.  The  "texture  parameters"  are  only  used  as  features  of 
which  we  hope  that  some  will  highlight  the  difference  between 
target  and  background.  Because  we  do  not  intend  to  measure  the 
texture  within  the  target,  the  parameters  are  useful  even  for 
small  targets  and  we  can  chose  an  arbitrary  displacement  (dx  = 

1,  dy  =1)  for  all  calculations  of  the  co-occurrence  matrix. 

The  results  for  each  texture  feature  can  be  converted  into  an 
image.  Figure  4  shows  the  texture  images  corresponding  to  the 
first  image  set  (IR  and  VIS)  of  sequence  MS01 . 


6-4 


IR  VIS 


Figure  4:  texture  images 


As  can  be  noticed  in  Figure  4,  the  vehicles  are  clearly  visible  in 
some  of  the  texture  images.  Hence  the  idea  to  combine  the 
texture  features  to  get  an  optimal  discrimination  between 
background  and  targets.  When  only  two  classes  are  involved,  as 
is  the  case  here  (targets/  background),  logistic  regression  offers 
an  appropriate  approach  [4]. 

In  the  learning  phase,  at  each  pixel  of  the  learning  image(s),  the 
texture  features  are  calculated  and  stored  in  a  table.  Then  the 
human  operator  interactively  indicates  the  bounding  rectangles 
surrounding  the  targets  in  the  learning  image(s)  and  a  column  is 
automatically  added  to  the  table  assigning  each  measurement  in 
the  table  to  either  class  0  (background),  when  it  corresponds  to 
an  image  pixel  that  falls  outside  the  bounding  rectangles,  or 
class  1  (targets)  when  it  is  inside  one  of  the  rectangles.  Logistic 
regression  is  then  used  to  find  a  combination  of  the  form  : 


bo +'£biFi(x.y) 

Px,y  (target  I  P)  =  g  '  - - 

'  1  '  bo+^biFi(x,y) 

1  +  e  1 

eq  1:  Logistic  Function 

in  which  pxy(  target  IF)  is  the  conditional  probability  that  a 
pixel  (x,y)  belongs  to  the  class  1  (target  class)  given  the  vector 
of  texture  parameters  F  at  the  given  pixel.  The  logistic 
regression  was  carried  out  using  Wald’s  forward  method.  In  this 
method,  at  each  step,  the  most  discriminant  feature  is  added  and 
the  significance  of  adding  it  to  the  model  is  verified.  This  means 
that  not  all  features  will  necessarily  be  included  into  the  model. 

Feature  level  fusion 

If  the  learning  images  are  representative  for  the  images  of  a 
given  sensor  type,  the  most  discriminating  features  for  each 
sensor  will  have  the  highest  weights  b,.  Therefore,  when  using 
the  same  weights  to  combine  the  feature  images  of  the 
remainder  of  the  database  into  new  images  using  equation  eq  1, 
targets  will  appear  as  local  maxima.  This  is  the  feature  level 
fusion. 

Sensor  Fusion 

The  sensor  fusion  step  combines  the  images  obtained  by  the 
feature  level  fusion  step.  In  the  feature-level-fused  images,  for 
each  sensor,  targets  appear  as  local  maxima.  Therefore  it  is 
possible  to  fuse  these  images  by  a  simple  multiplication.  In  the 
new  images  the  targets  will  still  appear  as  local  maxima. 

Region  Growing  around  Local  Maxima 

In  the  sensor-fused  image  the  local  maxima  will  correspond  to 
likely  target  positions.  To  detect  the  targets  it  is  thus  necessary 
to  find  these  local  maxima.  A  region  growing  procedure  around 
the  maxima  is  then  used  to  incorporate  available  prior 
knowledge  about  possible  target  size  and  aspect  ratio. 

Local  Maxima 

The  detection  of  local  maxima  is  based  on  a  succession  of 
morphological  operations  [11,  12],  The  basic  operator  is  a 
dilation  with  a  2  by  2  structuring  element. 

Region  Growing 

To  incorporate  any  available  prior  knowledge  about  the 
possible  range  of  target  size  or  aspect  ratio,  a  region  growing 
procedure  is  used.  The  initial  regions  for  the  region  growing  are 
the  local  maxima  in  the  image.  Surrounding  pixels  are  added  to 
these  regions  as  long  as  their  grey  level  differs  less  than  a  given 
threshold  from  the  value  at  the  local  maxima.  If  the  region 
becomes  too  large  it  is  discarded.  If  the  region  growing  of  a 
given  region  stops  before  it  reaches  the  upper  size-limit,  the 
other  constraints  are  checked.  If  a  constraint  is  not  satisfied,  the 
region  is  discarded. 

Clutter  rejection 
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To  reduce  the  number  of  false  targets,  a  simple  clutter  rejection 
scheme  was  implemented.  A  target  is  only  declared  if  it  was 
present  in  at  least  7  out  of  10  preceding  images.  More  clever 
tracking  methods  [13,  14]  could  be  used,  but  because  our  main 
interest  is  the  exploration  of  data  fusion  methods,  we  did  not 
implement  this  yet. 

Modes  of  operation 

The  presented  approach  for  the  detection  of  targets  in  single 
images  allows  us  to  experiment  with  different  levels  of  fusion. 
The  target  detection  can  be  performed  on  the  feature-level-fused 
images  of  each  separate  sensor  (mode  Ml). 

The  second  mode  (M2)  combines  the  feature-level-fused  images 
from  all  sensors  using  the  sensor  fusion  step  described  above. 

In  the  third  mode  (M3),  the  logistic  regression  is  applied  to  a 
superset  of  texture  features,  i.e.  the  feature-level-fused  image  is 
obtained  by  applying  the  logistic  function  to  the  set  of  features 
obtained  from  all  sensors.  This  is  only  possible  if  the  images 
from  all  sensors  are  registered. 

A  subdivision  of  modes  I  and  2  can  be  made  according  to 
whether  the  learning  images  used  were  acquired  from  the  same 
sensor  (i.e.  the  LW,  SW  and  TV  sensors  for  sequences  MS04- 
05)  or  from  the  same  generic  class  of  sensors  (i.e.  using  images 
from  the  LW  and  TV  sensor  of  sequence  MS01  to  yield  the 
weights  for  respectively  all  infrared-like  sensors  and  all  visual 
sensors). 


MC  - 


'L'Lcvj) 

j  i*j 

XXa/j) 


j  ‘ 


This  is  calculated  for  both  the  temporal  MClemp  and  for  the 
spatial  co-occurrence  matrix  MC  .  Sensor  motion  is  said  to  be 
present  if 


MClenlp  MCspat 


MC, 


>  0.005 


spat 


5.  MOVING  TARGET  DETECTION  (MTD  Module) 

The  second  part  of  the  algorithm  focusses  on  the  detection  of 
moving  targets.  In  order  to  detect  moving  targets,  any  sensor 
motion  needs  to  be  detected  and  its  effects  compensated  first. 
Then,  in  a  temporal  fusion  step,  preceding  images  can  be 
warped  onto  the  current  one.  Moving  objects  will  appear  as  a 
difference  between  the  original  image  and  the  warped  ones. 


Figure  5:  Detection  of  sensor  motion 

In  Figure  5  the  spatial  and  temporal  co-occurrence  matrix  are 
shown.  The  upper  images  show  the  matrices  for  a  part  of  a 
sequence  where  no  sensor  motion  was  present.  The  lower 
images  show  an  example  of  both  matrices  calculated  in  a  part  of 
the  same  sequence  where  the  sensor  was  moving. 

Motion  Estimation 


Detection  of  sensor  motion 

The  detection  of  sensor  motion  is  again  based  on  co-occurrence 
matrices.  This  time  the  co-occurrence  matrix  is  calculated 
between  an  image  and  the  preceding  one  (temporal  co¬ 
occurrence  matrix). 

C%fy/*{i,j)  =  P(G(x,y,t)  =  i  \G(x+ebc,y  +  dy,t  +  dt)  =  j)lf 

no  sensor  motion  occurred  between  the  two  images,  ideally,  for 
dx=dy=0  (i.e.  no  spatial  displacement),  all  non-zero  elements  of 
the  temporal  co-occurrence  matrix  should  lie  on  the  diagonal. 
However,  due  to  noise,  there  will  be  a  small  spread  along  the 
diagonal.  If  one  calculates  the  spatial  co-occurrence  matrix  for  a 
small  displacement,  the  spread  along  the  diagonal  is  due  to  noise 
and  to  the  fact  that  the  image  is  not  homogeneous.Therefore, 
when  comparing  this  spatial  co-occurrence  matrix  with  the 
temporal  co-occurrence  matrix,  the  spread  along  the  diagonal  is 
expected  to  be  the  largest  in  the  former  one  if  no  motion 
occurred  between  the  two  images  that  were  used  to  calculate  the 
temporal  co-occurrence  matrix.  When  motion  is  present,  the 
spread  along  the  diagonal  quickly  becomes  larger.  The 
measurement  we  used  to  detect  sensor  motion  is  based  on  the 
percentage  of  off-diagonal  points  in  both  co-occurrence 
matrices: 


If  sensor  motion  is  detected,  we  need  to  estimate  it  and 
compensate  its  effects  on  the  images.  Because  it  is  known  that 
the  sensor  is  stationary  and  can  only  do  a  pan  or  tilt,  the 
corresponding  motion  in  the  image  will  consist  of  a  uniform 
translation.  The  motion  is  estimated  by  searching  for  the 
translation  that  optimises  the  correlation  for  a  few  horizontal 
and  vertical  lines.  If  the  sensor  is  mounted  in  a  moving  vehicle 
or  no  apriori  knowledge  about  the  type  of  sensor  motion  is 
known,  methods  based  on  the  model  of  a  moving  rigid  planar 
patch  [15]  or  optical  flow  techniques  can  be  used  [16]. 

Detection  of  moving  targets 

Once  the  sensor  motion  is  estimated,  preceding  images  are 
warped  onto  the  current  one.  Then  the  original  image  is 
subtracted  from  the  warped  ones.  If  a  moving  object  is  present 
in  the  scene,  we  should  find  a  large  value  at  its  position.  The 
resulting  images  after  subtraction  are  therefore  thresholded  and 
objects  with  acceptable  size  and  aspect  ratio  are  selected  using  a 
region  growing  procedure.  Tracking  is  used  to  get  the  target  list. 
Figure  6  shows  the  result  of  subtracting  the  original  image  from 
the  warped  ones. 
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Imagc(I)  -  Imaged-])  Imagc(I)  -  Image(I-3)  Inugc(l)  -  Imugc(I-S) 


6.  DECISION  FUSION 


The  two  parts  of  the  algorithm  each  behave  as  experts  indicating 
the  possible  position  of  targets  in  the  scene.  The  final  decision  is 
reached  by  fusing  the  results  of  these  experts.  Because  each 
expert  only  provides  a  binary  decision  -  i.e.  either  a  target  is 
present  or  it  is  not  -  the  decision  fusion  is  implemented  as  a 
weighted  “k  out  of  N”  voting-rule[l,  17].  The  weights 
attributed  to  the  decisions  of  each  expert  can  depend  upon 
several  considerations. 

For  the  detection  of  moving  targets,  each  single  sensor  acts  as 
an  expert.  For  the  detection  of  targets  in  single  images,  the 
decision  was  made  after  fusing  the  feature  images  from  all 
sensors  in  Modes  M2  and  M3.  In  Mode  Ml,  results  from  each 
separate  sensor  are  passed  to  the  decison  level  fusion  step.  The 
weights  to  be  attributed  to  the  decision  of  each  expert  will  need 
to  be  adapted  accordingly. 

The  weights  may  also  depend  on  the  type  of  scenario.  In  the 
surveillance  scenario,  the  primary  expert  is  the  motion  detector, 
whereas  in  the  second  scenario  (reconnaissance),  both  types  of 
expert  are  equivalent. 

7.  RESULTS  AND  DISCUSSION 


In  this  section  the  results  for  the  two  parts  of  the  algorithm  are 
presented  first.  Then  some  results  of  decision  fusion  are 
presented. 

Results  of  TDSI  Module 

Results  of  the  feature-level  fusion 

For  the  feature  level  fusion  a  learning  image  set  needs  to  be 
identified.  For  the  so-called  “generic  sensor  case”(GSC),  the 
learning  image  set  consists  of  the  first  Infrared  (LW)  and  Visual 
image  of  MS01  on  which  the  targets  appear. 

For  the  “sensor  kind  case”(SKC)  three  sets  of  learning  images 
were  identified.  For  sequences  MS01-MS03  they  are  the  same 
as  the  ones  used  in  the  “generic  case”.  For  sequences  MS04- 
MS05  the  first  multi-sensor  image  set  of  MS04  is  used,  yielding 
weights  for  LW-.SW-  and  TV-type  sensors.  For  sequences 
MS06  the  first  image  set  of  MS06  is  used.  For  the  fusion  of  the 
superset  of  features  (SSF)  the  learning  images  were  the  same  as 
for  the  “sensor  kind  case”.  The  following  table  presents  the 
weights  obtained  for  MS04-MS05. 


Case 

Sens 

bO 

bl 

b2 

b3 

b4 

b5 

b6 

GSC 

LW 

-15 

2.4 

4.7 

0 

0.01 

SW 

TV 

-18 

0 

0 

4.7 

5.6 

2.6 

0 

SKC 

LW 

-18 

0 

0.4 

0 

6.3 

4.8 

-0.008 

SW 

-18 

0 

0 

0 

7.0 

10 

-0.06 

TV 

-19 

0 

-0.06 

4.6 

7.6 

3.5 

-0.004 

SSF 

LW 

-31 

4.9 

0 

0 

8.9 

0 

0 

SW 

0 

0 

0 

0 

0 

0 

-0.05 

TV 

0 

0 

0 

0 

6.8 

8.8 

-0.06 

Table  2:  Weights  obtained  by  logistic  regression  for 
sequences  MS04  and  MS05. 


Results  for  the  single  sensors  (Ml ) 

For  each  sequence,  the  probability  of  detection  and  the  average 
number  of  false  targets  per  image  is  given  for  both  the  “Generic 
Sensor  Case”  and  the  “Sensor  Kind  Case”.  Please  note  that  for 
the  first  three  sequences,  the  two  cases  are  identical  because  the 
“generic  sensor  set”  is  the  set  of  sensors  that  were  used  to 
acquire  these  sequences.  For  sequences  MS04  -  MS06  the 
results  obtained  for  the  SKC  case  are  slightly  better  than  those 
obtained  for  the  GSC  in  most  cases.  However,  the  inverse  is  true 
for  MS06LW.  This  is  due  to  the  choice  of  the  learning  image  in 
the  SKC  case.  The  learning  image  is  the  first  image  on  which 
the  target  appears.  In  the  infrared  image,  the  target  happens  to 
be  white  on  a  light  grey  background  (clouds)  and  is  very 
difficult  to  see.  Because,  in  a  part  of  the  sequence,  its 
background  becomes  a  clear  sky  (dark),  the  weights  are  no 
longer  appropriate  and  performance  drops. 


Sequence 

Sensor 

GSC 

SKC 

Pd 

I  Nft 

Pd  | 

Nft 

MS01 

LW 

85 

10 

85 

10 

TV 

32 

4 

32 

4 

MS02 

LW 

0 

2 

0 

2 

TV 

0.5 

1.4 

0.5 

1.4 

MS03 

LW 

84 

11 

84 

11 

TV 

21 

16 

21 

16 

MS  04 

LW 

97 

16 

98 

13 

SW 

43 

2 

63 

6 

TV 

98 

14 

93 

13 

MS05 

LW 

81 

12 

99 

14 

SW 

0 

1.16 

9 

4 

TV 

51 

14 

98 

11 

MS06 

LW 

50 

27 

31 

7 

RD 

89 

6 

94 

2.5 

GR 

96 

0.4 

73 

2.4 

BL 

90 

0.25 

82 

0.57 

Table  3:  Results  for  single  sensors 


In  MS02  targets  disappear  in  the  large  number  of  false  targets 
caused  by  noise.  They  are  only  sporadically  detected  by  the  first 
part  of  the  algorithm  and  rejected  by  the  clutter  rejection  stage. 

Results  after  sensor  fusion  (M2) 

The  following  table  presents  the  results  after  the  sensor  fusion 
step. 
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Sequence 

GSC 

SKC 

Sequence 

Pd 

Nft 

Pd 

Nft 

MS01 

83 

3 

83 

3 

MS02 

20 

5 

20 

5 

MS03 

16 

16 

16 

16 

MS04 

95 

2 

98 

0.07 

MS05 

94 

0.2 

96 

0.15 

MS06 

23 

7 

47 

0.8 

Table  4:  Results  after  sensor  fusion 


Results  using  superset  of  features  (M3) 


Sequence 

Pd 

Nft 

MS01 

88 

10 

MS02 

0.5 

1.7 

MS03 

93 

31 

MS04 

0.4 

12 

MS05 

0 

20 

MS06 

40 

6 

Table  5:  Results  of  superset  of  features 


Results  of  MTD  Module 

For  the  moving  target  detection  a  threshold  is  defined  as  the 
number  of  subsequent  images  in  which  the  target  is  detected. 
The  maximal  number  of  subsequent  images  is  9. 

The  following  tables  present  the  results  as  a  function  of  the 
threshold  T.  Moving  targets  were  only  found  in  sequences 
MS01-MS03  and  in  MS06.  For  MS06  results  for  the  three 
visual  components  were  very  similar,  therefore  only  the  red 
component  is  shown. _ _ 


T 

MS01 

MS  02 

LW 

TV 

LW 

TV 

Pd 

Nft 

Pd 

Nft 

Pd 

Nft 

Pd 

Nft 

1 

88 

0.06 

48 

0.87 

30 

1.6 

39 

0.48 

2 

82 

0.03 

47 

0.7 

25 

0.37 

36 

0.29 

3 

63 

0.01 

45 

0.5 

19 

0.16 

32 

0.19 

4 

53 

0 

44 

0.15 

12 

0.06 

23 

0.14 

5 

52 

0 

44 

0.02 

9 

17 

0.10 

6 

52 

0 

42 

0.02 

4 

0.02 

9 

0.06 

7 

51 

0 

41 

0 

i 

0.01 

4 

0.03 

8 

0 

0 

0 

0 

0 

0 

0 

0 

Table  6:  Results  of  MTD  for  sequences  MS01  and  MS02 


T 

MS03 

MS06 

LW 

TV 

LW 

RD 

Pd 

Nft 

Pd 

Nft 

Pd 

Nft 

Pd 

Nft 

1 

77 

0.02 

55 

0.03 

7 

1.1 

81 

0.19 

2 

74 

0.02 

44 

0.02 

4 

0.62 

81 

0.15 

3 

73 

0.01 

38 

0.02 

2 

0.26 

80 

0.15 

4 

73 

0.01 

24 

0.02 

0 

0.12 

75 

0.11 

5 

71 

0.01 

6 

0.01 

0 

0.04 

65 

0.09 

6 

66 

0.01 

0 

0.01 

0 

0 

51 

0.06 

7 

58 

0.01 

0 

0.01 

0 

0 

32 

0.04 

8 

9 

0 

0 

0 

0 

0 

4 

0 

Table  7:  Results  of  MTD  for  sequences  MS03  and  MS06 
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Figure  7:  Probability  of  detection  for  MTD  vs.  threshold 
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Figure  8:  Average  number  of  False  Targets  per  image  for 
MTD  vs.  threshold 

In  Figure  7  and  Figure  8  it  can  be  noted  that  by  setting  the 
threshold  to  3  the  number  of  false  targets  is  greatly  reduced 
while  the  probability  of  detection  is  hardly  affected.  Therefore 
we  will  use  the  threshold  3  in  the  decision  fusion. 

Another  remark  is  that  in  sequences  where  moving  targets  are 
detected,  the  MTD  module  outperforms  the  TDSI  module  (cf. 
Table  3,  Table  6  and  Table  7) 

Results  after  decision  fusion 


In  sequences  corresponding  to  scenario  1  (surveillance)  the 
motion  detector  is  the  primary  expert.  In  the  decision  fusion  we 
should  therefore  attribute  the  highest  weight  to  it. 


6-8 


In  the  reconnaissance  scenario,  the  motion  detector  is  equivalent 
to  the  target  detector  in  single  images  and  both  experts  should 
have  the  same  weight. 

The  “K  out  of  N”  decision  fusion  only  accepts  a  target  if  the 
number  of  experts  Nexperls  that  have  detected  it  is  above  a 

certain  threshold  T.  Special  cases  for  this  threshold  are: 

•  AND:  T=Nexperts 

•  OR:  1=1 

•  Majority  voting:  T  =  — +  j 

The  following  tables  present  the  results  for  both  scenarios  at 
different  thresholds.  In  Table  8  the  results  for  sequences  MS01- 
MS03  are  shown  after  fusion  of  the  outcomes  for  each 
individual  sensor(mode  Ml).  In  Table  9  the  results  of  fusing  the 
sensor-fusion  results  with  the  motion  data  are  shown  for  the 
same  three  sequences  (M2).  Please  note  that  sequences  MS01- 
MS3  were  acquired  using  two  sensors,  the  number  of  experts  is 
2  for  the  TDSI  module  (in  mode  M 1 )  and  2  for  the  MTD 
module.  In  the  decision  fusion  both  parts  of  the  algorithm  have 
the  same  weight.  In  mode  M2  only  one  expert  is  available  for 
the  TDSI  module.  However,  it  is  counted  twice  to  ensure  that 
both  parts  of  the  algorithm  have  the  same  weight  in  the  final 
decision. 

Table  10  shows  the  results  of  the  decision  fusion  for  MS01- 
MS03  using  the  super-set  of  features  (mode  M3). 

Table  1 1  shows  the  results  of  the  decision  fusion  for  sequence 
MS06  using  the  outcomes  of  the  TDSI  module  applied  to 
individual  sensors  (mode  Ml)  and  the  results  using  the 
outcomes  of  the  sensor  fusion  step.  For  both  modes,  the  results 
are  given  for  the  “generic  sensor  case”  and  the  “sensor  kind 
case”.  Please  note  that,  as  in  sequence  MS06,  we  have  4  sensors 
(LW,RD,GR  and  BL),  there  are  8  independent  experts. 

In  sequences  MS04  and  MS05  all  targets  are  stationary  and  the 
MTD  module  does  not  report  any  targets.  Therefore  the  decision 
fusion  step  is  not  necessary. 


T 

MS01 

MS  02 

MS03 

Pd 

Nft 

Pd 

Nft 

Pd 

Nft 

1 

96 

20 

36 

2 

85 

11 

2 

66 

3 

16 

0.7 

73 

1 

3 

54 

0.86 

3 

0.24 

39 

0.5 

4 

30 

0.27 

3 

0.18 

0 

0.01 

Table  8:  MS01-MS03:  Results  of  decision  fusion  using 
outcomes  from  individual  sensors  (mode  Ml) 


T 

|  MS01 

!  MS02  j 

1  MS03  I 

Pd 

Nft 

Pd 

Nft 

Pd 

Nft 

1 

65 

3 

54 

5 

79 

20 

2 

49 

0.9 

17 

0.4 

63 

0.8 

3 

3 

0.11 

3 

0.1 

19 

0.18 

4 

0 

0.01 

0.5 

0.01 

6 

0.01 

Table  9:  MS01-MS03:  Results  of  decision  fusion  using 
outcomes  from  sensor  fusion  step  (mode  M2) 


T 

I  MS01  j 

i  MS  02 

MS03  1 

Pd 

Nft 

Pd 

Nft 

Pd 

Nft 

1 

95 

11 

36 

2 

90 

28 

2 

58 

0.9 

16 

0.46 

67 

1.98 

3 

36 

0.31 

3 

0.09 

43 

0.53 

4 

3 

0.05 

0.5 

0.01 

7 

0.05 

Table  10:  MS01-MS03:  Results  of  decision  fusion  using 
outcomes  of  the  super  set  of  features  (mode  M3) 


i 

MS06 

MS06 

MS06SF 

MS06SF 

MS06 

■ 

(GSC) 

(SKC) 

(GSC) 

(SKC) 

(SSF) 

■ 

Pd 

Nft 

Pd 

Nft 

Pd 

Nft 

Pd 

Nft 

Pd  Nft 

1 

98 

34 

98 

13 

98 

98 

84  6.5 

2 

98 

3 

94 

2.5 

69 

64 

55  1 

3 

86 

1.4 

73 

1.1 

29 

19 

6  0.2 

4 

58 

0.7 

32 

0.7 

0.9 

0.9 

4  5.9 

Table  1 1 :  MS06:  Results  of  decision  fusion  using  the 
GSC  and  SKC  for  individual  sensors  and  after  sensor 
fusion  and  using  the  super  set  of  features  (SSF). 

Discussion 

For  the  discussion,  the  fusion  method  that  gives  the  best  results 
will  be  identified  for  each  sequence  separately.  Then  these  “best 
results”  will  be  analysed  as  a  function  of  some  of  the  properties 
of  the  sequences.  The  notion  of  best  results  depends  on  the 
application.  For  some  applications  the  probability  of  detection  is 
the  critical  factor  while  for  others  the  ratio  between  probability 
of  detection  and  number  of  false  targets  has  to  be  maximised. 

To  identify  the  “best  fusion  method”  for  each  sequence,  we  have 
chosen  the  latter  criterion  (Pd/Nft).  In  the  previous  tables,  these 
best  result  are  highlighted  with  italic  letters  on  a  grey 
background.  Please  note  that,  for  sequences  where  moving 
targets  are  present,  the  MTD  algorithm  used  on  a  single  sensor 
sometimes  gives  a  higher  ratio  Pd/Nft  than  any  of  the  fusion 
results.  In  all  cases  fusion  will  however  increase  the  probability 
of  detection. 

Best  results  per  sequence 
MS01: 

For  MS01  the  best  results  are  obtained  after  decision  fusion  of 
the  results  obtained  with  the  super-set  of  features  (TDSI-Mode 
M3)  and  the  results  from  the  moving  target  detection. 

MS02: 

For  MS02,  the  results  of  the  TDSI  module  for  single  sensors 
(mode  Ml)  and  for  the  super-features  (mode  M3)  are  very  poor. 
The  targets  are  completely  lost  in  the  noise  and  rejected  by  the 
clutter  rejection  algorithm.  For  mode  M2,  results  are  slightly 
better.  This  is  because  the  targets  are  enhanced  by  multiplying 
the  featurc-level-fused  images  while  noise  is  tampered.  The 
moving  target  detection  gives  better  results.  In  the  decision 
fusion,  the  best  results  are  obtained  by  fusing  TDSI-mode  M2 
with  the  MTD  results. 


MS03: 

For  this  sequence  the  best  results  are  obtained  by  the  decision 
fusion  of  the  results  obtained  by  the  TDSI  module  for  single 
sensors  and  the  MTD  outcomes. 
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MS04  and  MS05: 

The  MTD  module  does  not  report  any  targets.  The  best  results 
are  obtained  by  mode  M2  of  the  TDSI  module. 

MS06: 

The  LW  and  TV  sensors  compete  for  the  detection  of  the 
targets.  The  SW  gives  less  good  results.  The  best  results  are 
obtained  after  the  decision  fusion  of  the  sensor  fused  data  from 
the  SKC  case.  It  is  interesting  to  note  that  the  difference 
between  the  results  of  the  SKC  and  the  GSC  case  is  very  small. 


Analysis  of  “best  results": 

We  will  now  try  to  explain  why  a  given  fusion  method  gives  the 
best  results  for  each  sequence.  We  will  try  to  correlate 
subjective  notions  of  image  quality  and  sequence  uniformity  to 
the  results. 

For  MS04  and  MS05  the  MTD  doesn’t  find  any  targets, 
therefore  it  makes  no  sense  to  perform  the  decision  fusion  of 
TDSI  results  with  MTD  results.  For  these  two  sequences  the 
LW  and  TV  sensors  give  images  that  have  a  similar  quality 
while  the  SW  sensor  gives  a  much  lower  contrast. 

For  MS01  both  sensors  provide  images  that  have  a  similar 
quality  and  the  sequence  is  uniform  (targets  have  the  same 
contrast  with  their  surroundings  throughout  the  sequence).  For 
MS02  the  images  obtained  from  the  LW  sensor  are  less  good 
than  those  obtained  from  the  TV  sensor. 

For  MS06  this  is  also  the  case.  For  MS03  the  target  is  hardly 
visible  in  the  beginning  of  the  sequence  while  its  contrast 
gradually  increases. 

We  can  make  the  following  statements: 

•  If  the  sequence  is  not  uniform,  using  M3  gives  bad  results 
because  the  test  image  is  not  characteristic  for  the  rest  of 
the  database  and  using  the  super-set  features  causes 
features  that  allow  detection  for  some  sensors  to  be 
discarded. 

•  If  the  sensors  are  equivalent  and  the  sequence  is  uniform, 
M3  gives  good  results. 

•  If  one  of  the  sensors  provides  images  of  lesser  quality,  both 
using  the  super-set  of  features  (M3)  and  the  single  sensors 
(Ml)  give  rise  to  too  many  false  targets  (due  to  noise).  The 
sensor  fusion  step  (M2)  however  reduces  the  potential  false 
targets  before  the  actual  target  detection. 

•  Because  a  simple  multiplication  is  chosen  for  the  sensor 
fusion  step,  the  method  based  on  sensor  fusion  (M2)  is  not 
applicable  if  one  of  the  sensors  is  not  operational. 


The  following  figure  presents  these  findings  graphically. 


At?  Ml  M2 


Although  these  results  seem  logical,  due  to  the  fact  that  we  only 
have  6  sequences  to  test  this  and  because  we  did  not  use  any 
objective  measurements  of  image  quality  or  sequence 
uniformity,  this  can  not  be  generalised. 

8.  CONCLUSIONS 

In  this  paper  the  use  of  data  fusion  at  several  levels  is  explored. 
The  method  was  tested  on  a  database  of  6  multi-spectral  image 
sequences.  The  approach  consists  of  two  main  parts.  The  first 
part  detects  targets  in  single  images  (TDSI  module)  while  the 
second  part  tries  to  detect  moving  targets  (MTD  module).  The 
motion  detection  is  performed  for  each  sensor  separately. 

The  TDSI  module  is  based  on  texture  features.  Several  texture 
features  are  calculated  in  each  point  of  the  images.  These 
features  are  combined  into  a  new  image  using  feature  fusion. 
For  the  TDSI  module  three  modes  of  operation  are  identified.  In 
Mode  Ml  the  detection  is  performed  for  each  sensor  separately. 
Mode  M2  performs  sensor  fusion  by  combining  the  images 
obtained  by  feature-level-fusion  from  each  sensor.  Mode  M3 
determines  the  feature-level-fused  images,  using  features  from 
all  sensors  (super-features). 

Decision  level  fusion  is  used  to  combine  the  results  of  the  two 
parts  of  the  algorithm. 

Results  show  that  the  MTD  module  is  very  efficient  when 
moving  targets  are  present.  In  the  TDSI  module,  the  different 
sensors  appear  to  be  quite  complementary:  in  some  sequences 
the  infrared  sensors  give  the  best  results  while  in  others,  the 
visual  sensor  outperforms  the  infrared  sensor. 

The  results  show  that  the  type  of  fusion  that  gives  the  best 
performance  varies  greatly  from  sequence  to  sequence.  The 
performance  is  influenced  by  the  presence  of  noise  fin  MS02), 
the  presence  of  a  less  performant  sensor  (in  MS02,  MS04  and 
MS05),  the  type  of  scenario  and  the  uniformity  of  the 
background.  Although  we  do  not  pretend  that  there  is  a  single 
optimal  fusion  paradigm  that  can  solve  all  possible  problems  in 
all  possible  cases,  for  our  test  sequences,  we  did  find  a  way  to 
chose  the  optimal  fusion  method  among  the  methods  we  tested, 
based  on  the  criteria  that  are  given  above. 

This  paper  has  presented  some  of  the  advantages  and/or 
disadvantages  of  using  data  fusion  on  different  levels. 

As  a  final  conclusion  one  could  state  that  different  kinds  of  data 
fusion  have  different  advantages  and  disadvantages  and  are 
therefore  suited  for  solving  different  kinds  of  problems. 
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Abstract 

(NU)  The  FGAN-FHP  MEMPHIS  radar 
transmits  simultaneously  at  35  GHz  and 
94  GHz.  Accordingly,  when  flown  in  a  side¬ 
looking  configuration,  closely  related  SAR 
images  can  be  generated  for  both  frequen¬ 
cies.  These  two  images  can  be  combined  for 
the  purpose  of  target  detection  and  discrim¬ 
ination. 

(NU)  It  is  analyzed  how  this  combination 
can  be  performed,  and  how  much  one  gains 
with  respect  to  speckle  reduction,  target-to- 
background  contrast,  detection  probability 
and  false  alarm  rate.  For  this  purpose,  a  sim¬ 
ple  algorithm  is  considered  that  consists  of 
a  threshold  based  prescreener  and  a  subse¬ 
quent  feature  based  discrimination  stage. 

(NU)  For  multi-look  images  that  are 
mapped  to  the  ground  it  is  found  that 
the  combination  of  two  different  frequen¬ 
cies  with  like  polarization  provides  better 
speckle  reduction  and  target-to-background 
enhancement  than  the  combination  of  two 
orthogonal  polarization  channels  at  the  same 
frequency.  The  detection/discrimination  re¬ 
sults  depend  on  how  precisely  the  two  im¬ 
ages  are  registered. 

1.  Introduction 

(NU)  During  the  last  years,  radar  imaging  has 
gained  more  and  more  importance  for  military 
applications.  In  multi-sensor  drones  for  surveil¬ 
lance  and  reconnaissance  applications  as  well  as  in 
dual  mode  seekers  where  radar  signals  have  to  be 
matched  to  IR  images,  it  is  necessary  to  process  the 
returns  of  the  radar  sensor  into  images.  Only  on 
the  basis  of  images  extended  high  value  targets  like 
bridges,  industrial  complexes,  bunkers,  command 
posts  or  others  can  be  detected  and  analyzed  and 
later  on  fused  with  the  corresponding  output  of  the 
IR  seeker,  or  matched  with  pregiven  patterns.  Also 
for  smaller  relocatable  targets  the  improvement  in 
resolution  leads  away  from  the  former  point  target 
treatment  to  imaging  methods  for  the  purpose  of 
detection  and  discrimination. 


(NU)  The  MEMPHIS  radar,  used  by  FGAN- 
FHP  for  airborne  measurements  in  a  side-looking 
SAR  configuration,  is  operated  simultaneously  at 
35  GHz  and  94  GHz.  Both  frequencies  are  derived 
from  a  common  system  reference.  All  system  pa¬ 
rameters  like  pulse  shape  and  duration,  PRF,  trans¬ 
mit  and  receive  polarization,  phase  code  for  high 
range  resolution  etc.  are  identical.  The  two  anten¬ 
nas  are  boresighted  to  the  the  best  possible  preci¬ 
sion.  A  detailed  description  can  be  found  in  [13]. 

(NU)  With  this  interesting  data  at  hand,  the 
question  arose  whether  by  combining  information 
from  these  two  physically  completely  independent 
frequencies,  an  improvement  in  image  quality,  tar¬ 
get/background  contrast  and  detection  algorithm 
performance  can  be  achieved.  As  a  basis  for  the 
analysis,  a  scene  is  chosen  containing  five  different 
relocatable  targets  (missile  launchers,  air  defence 
units  )  and  several  vehicles  of  opportunity  (trucks, 
vans  etc.).  These  targets  are  located  in  a  grassy 
area  with  scattered  isolated  trees  and  bushes,  sur¬ 
rounded  by  a  wooded  area  with  roads  and  several 
buildings.  The  detection  algorithm  is  designed  to 
discriminate  the  relocatable  targets  from  the  natu¬ 
ral  background  and,  if  possible,  also  from  the  build¬ 
ings. 

(NU)  Section  2  gives  a  description  of  the  data 
that  are  used  for  the  analysis,  and  of  various  ways 
how  different  SAR  images  can  be  registered  to  each 
other.  In  section  3,  the  effect  of  combining  different 
images  on  speckle  reduction  and  target/background 
contrast  enhancement  is  described  in  detail.  The 
influence  of  image  combination  on  detection  and 
discrimination  algorithm  performance  is  analyzed 
in  section  4.  The  detection  results  are  analyzed  and 
summarized,  leading  to  the  conclusions  in  section 
5. 

2.  Registration  of  images 

The  basic  element  of  every  SAR  image  is  the  local 
range-Doppler  map  (RDM),  i.e.  the  outcome  of  the 
Doppler  Fourier  transform.  This  RDM  is  projected 
onto  a  ground  map  with  predefined  pixel  spacing, 
thereby  making  use  of  all  motion  and  aircraft  at¬ 
titude  information.  Consecutive  RDMs  may  cause 
several  returns  to  be  projected  to  the  same  pixel 
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Fig.  1.  SAR  image  of  relocatable  target  area  at  35  GHz  (left)  and  94  GHz,  polarization  LR 


0.55 


Ax 


Fig.  2.  Cross  correlation  coefficient  between  35  GHz  and 
94  Ghz  scene 


images,  one  has  to  make  two  decisions.  First,  does 
one  want  to  work  with  local  RDMs  or  with  ground 
maps?  Second,  does  the  resulting  image  have  to 
contain  (multichannel)  amplitude  plus  phase  or  am¬ 
plitude  only? 

(NU)  When  working  with  local  RDMs  one  has 
to  take  into  account  the  properties  of  the  Doppler 
Fourier  transform  (DFT  or,  most  often,  FFT).  One 
can  use  any  aperture  time  ‘T’  up  to  the  maximum 
possible  to  get  an  according  Doppler  resolution 

_  A  _  XR 
~  2/3  ~~  2vT 

(v=flight  velocity,  R=slant  range,  A=wave  length, 
/3=change  in  aspect  angle).  If  one  processes  !N’ 
consecutive  pulses  then  the  resulting  unambiguous 
range  ‘R y’  of  the  FFT  is  given  by 


on  the  ground.  If  one  eliminates  this  overlap  the 
final  product  is  a  single-look  image  which  can  still 
be  complex,  i.e.  contain  phase  information.  If  one 
averages  the  signals  within  each  pixel  the  result  is 
a  multi-look  image  which  normally  contains  only 
intensity  (one  way  to  retain  a  certain  phase  infor¬ 
mation  is  Stokes  averaging  [2]). 

(NU)  Therefore,  before  combining  two  different 


Ru  =  NA  = 


XR  XR  XR 

_  — -  _  __  PRF 

2 v(T/N)  2vAT  2v  ' 


If  one  wants  to  obtain  equal  Doppler  resolution  one 
has  to  fulfill  the  condition  A/T=const.,  hence  the 
processing  times  ‘T’  must  have  the  ratio  94/35  = 
2.687  &  8/3.  As  both  radars  are  operated  with  the 
same  PRF  this  means  that  the  DFT  lengths  are 
different  by  this  ratio,  and  this  again  means  that 
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Fig.  3.  combined  SAR  image  35+94  GHz  | 
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one  can  use  an  FFT  only  for  one  case,  but  a  general 
DFT  for  the  other.  This  can  be  very  processing  time 
consuming  and  therefore  is  impractical. 

(NU)  The  most  natural  thing  would  be  to  keep 
A  •  PRF= const.,  which  would  mean  equal  unam¬ 
biguous  range.  In  addition,  one  can  obtain  equal 
Doppler  resolution  by  choosing  the  same  FFT 
length  which  also  facilitates  the  processing.  On 
the  other  hand,  as  the  aperture  lengths  and  times 
would  differ  by  the  factor  94/35,  this  means  that 
multi-look  images  cannot  have  the  same  degree  of 
integration  for  both  frequencies.  However,  as  the 
MEMPHIS  radar  has  one  and  the  same  PRF  for 
35  GHz  and  94  GHz,  the  only  way  to  fulfill  the 
above  condition  would  be  to  thin  down  the  original 
PRF  by  using  e.g.  each  8th  pulse  for  35  GHz  and 
each  third  pulse  for  94  GHz.  This  is  not  realistic, 
of  course,  because  one  would  throw  away  valuable 
data  and,  above  all,  the  effective  PRF  would  be¬ 
come  so  low  that  the  unambiguous  range  is  smaller 
than  the  beamwidth  between  nulls. 

(NU)  The  simplest  way  to  proceed,  therefore,  is 
to  perform  adequate  FFTs  for  each  frequency  in¬ 
dependently  and  match  the  resolution  cell  sizes  by 
means  of  interpolation.  In  the  case  of  equal  FFT 
lengths,  for  example,  3  Doppler  cells  at  35  GHz 
would  have  approximately  the  same  extent  as  8 


Fig.  4.  Amplitude  and  CFAR  profile  for  35  GHz  (dot¬ 
ted),  94  GHz  (dashdot)  and  35+94  GHz  (solid) 


cells  at  94  GHz.  Consequently,  8  cells  with  values 
bi. . .  bs  would  be  constructed  out  of  the  original  3 
cells  with  values  aj.,  a2  and  a3,  using 

5l  =  &2  —  al 

,  2  1 

03  —  +  -o2 

64  =  65  =  a2 

,  1  2 

06  = 

br  —  bs  =  a  3 

either  in  real  (intensity)  or  in  complex  space.  The 
resulting  interpolated  35  GHz  image  would  then  be 
matched  to  the  94  GHz  image  with  the  same  reso¬ 
lution  and  the  same  number  of  Doppler  cells  (see 
below) . 

(NU)  The  methods  described  so  far  are  adequate 
for  any  detection  algorithm  that  works  on  individ¬ 
ual  RDMs  or  on  small  ensembles  of  consecutive 
RDMs.  If,  on  the  other  hand,  one  works  with  radar 
maps  of  a  somewhat  larger  extent  that  were  created 
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by  means  of  motion  compensation  and  projection 
on  a  ground  map,  then  the  case  is  much  easier  as 
the  two  radars  are  mounted  on  the  same  platform. 
Here,  the  result  of  the  processing  are  two  maps  with 
the  same  resolution  and  without  (or  with  identical) 
geometrical  distortions. 

(NU)  As  there  will  always  be  a  certain  bore¬ 
sighting  error  between  the  35  GHz  antenna  and  the 
94  Ghz  antenna,  the  next  step  for  both  RDM  or 
ground  map  is  the  matching  of  the  two  contribu¬ 
tions  by  means  of  correlation.  For  this  purpose  it  is 
usually  sufficient  to  select  only  a  part  of  the  com¬ 
plete  map,  containing  a  number  of  dominant  scat¬ 
tered,  to  reduce  the  computational  effort. 

(NU)  The  following  analysis  was  done  with  multi¬ 
look  ground  maps  consisting  of  two  polarization 
channels.  The  measurements  used  for  this  purpose 
were  performed  with  circular  polarization.  A  de¬ 
tailed  description  can  be  found  in  [1]  and  [10].  Us¬ 
ing  circular  transmit  polarization  leads  to  a  nat¬ 
ural  splitting  of  the  receive  signal  into  one  part 
that  underwent  an  odd  number  of  reflections  (flat 
plate  or  trihedral  corner),  and  another  part  that 
underwent  an  even  number  of  reflections  (dihedral 
type).  Therefore,  we  use  the  terms  ‘odd’  (LR,  or 
LHC  transmit  and  LHC  receive)  and  ‘even’  (LL) 
channel  in  the  following. 

(NU)  A  polyphase  coded  wave  form  was  used 
with  a  bandwidth  of  100  MHz  resulting  in  a  high 
range  resolution  (HRR)  of  about  1.5  meter,  the 
sampling  interval  was  5  ns  providing  a  HRR  cell 
spacing  of  0.75m  .  The  nominal  depression  angle 
was  17.5°,  at  a  slant  range  of  800  m.  The  3-dB  two- 
way  beamwidths  of  the  lens  antennas  were  1°  (2.7°) 
in  azimuth  and  12°  (15°)  in  elevation  for  94 GHz 
(35  GHz).  The  Doppler  processing  was  performed 
in  such  a  way  that  the  along-track  resolution  was 
1.0  meter.  The  resolution  cells  were  projected  on  a 
ground  map  with  a  grid  size  of  1  m  by  1  m,  result¬ 
ing  in  a  number  of  5  to  6  complex  data  values  per 
pixel  ([14],  [15]). 

(NU)  The  scene  (cf.  fig.l)  is  about  800  m  long  and 
500  m  wide.  For  the  correlation  a  region  of  length 
100  m  and  width  140  m  containing  4  of  the  relocat¬ 
able  targets  was  chosen.  The  correlation  was  per¬ 
formed  over  the  intensity  values  (in  dB)  of  the  LR 
channel.  A  relatively  sharp  maximum  was  found 
with  an  offset  of  2  pixels  both  in  ‘x’  and  ‘y’  di¬ 
rection,  the  correlation  coefficient  at  the  maximum 
being  pm 

ax  — 523  (cf.  fig. 2).  As  the  images  (multi¬ 
look  two-channel  amplitude-only)  were  given  in  dB- 
scale,  also  the  combination  of  the  two  data  sets  was 
done  by  taking  the  mean  (in  dB)  of  the  two  contri¬ 
butions  within  each  pixel.  This  corresponds  to  the 
geometric  mean  of  the  return  powers.  One  could  as 


well  use  the  arithmetic  mean,  but  this  would  have 
meant  to  linearize  all  values  first,  then  average  and 
transform  back  to  dB-space,  hence  a  much  more  te¬ 
dious  procedure.  The  final  product  are  two-channel 
(odd  and  even)  intensity  images. 

3.  Target  /  background  contrast 
enhancement 

Fig. 3  shows  the  result  of  combining  the  35  GHz 
and  the  94  Ghz  image.  When  comparing  this  to  the 
original  images  one  notices  essentially  three  effects: 
the  speckle  appears  reduced,  i.e.  the  image  looks 
smoother,  a  slight  loss  in  focus  occurred,  and  the 
shadows  are  more  pronounced,  i.e.  the  contrast  was 
enhanced. 

(NU)  When  looking  at  the  smeckle  reduction,  one 
sees  a  general  dilemma.  For  a  precise  registration  of 
the  two  images,  a  high  correlation  is  needed.  On  the 
other  hand,  additional  information  is  gained  only, 
when  the  images  are  independent  which  means  that 
their  correlation  is  low.  For  the  assessment  of  im¬ 
age  smoothing,  two  subareas  were  chosen  within  the 
image.  The  first  one  contains  mixed  clutter,  mainly 
grass  with  scattered  bushes,  the  second  one  is  a  for¬ 
est  of  deciduous  trees.  As  a  measure  of  smoothness 
we  consider  the  standard  deviation  of  the  reflectiv¬ 
ity  calculated  in  dB-space: 


channel 

mixed  clt. 

decid.  forest 

LR 

3.77 

4.21 

35  GHz 

LL 

4.00 

4.46 

LR+LL 

3.54 

4.00 

LR 

3.85 

4.41 

94  GHz 

LL 

3.54 

4.12 

LR+LL 

3.51 

4.11 

LR 

3.29 

3.78 

35+94GHz 

LL 

3.25 

3.79 

LR+LL 

3.14 

3.65 

Table  1:  Reflectivity  standard  deviations 
(dBm2/m2) 

(NU)  For  comparison,  this  table  contains  also 
values  for  the  case  when  both  orthogonal  receive 
channels  are  combined  (summing  the  linearized 
backscatter  powers  and  transforming  the  result 
back  to  dB).  One  sees  that  the  combination  LR+LL 
results  in  a  smoothing  with  respect  to  LR  or  LL 
alone,  that  the  combination  35+94  GHz,  however, 
provides  a  more  pronounced  smoothing.  This  is  in 
agreement  with  the  correlation  coefficient  between 
the  LR  and  the  LL  channels  which  is  0.583  for 
35  GHz  and  0.696  for  94  GHz,  respectively. 

(NU)  The  slight  loss  in  focus  is  due  to  the  slight 
boresight  error  between  the  two  antennas.  Indepen- 
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dent  measurements  show  an  azimuth  misalignment 
of  0.4°  which  means  that  the  crab  angle  correction 
must  be  different  for  both  images.  The  correlation 
method  when  it  is  applied  to  the  complete  map 
gives  the  best  results  of  course  in  the  correlation 
region.  Moreover,  an  optimal  registration  should  al¬ 
low  not  only  for  linear  offsets  but  also  for  a  slight 
rotation. 

(NU)  The  deepening  of  the  shadows  is  due  to 
the  fact  that  the  pertinent  cells  contain  only  noise 
which  ideally  has  zero  correlation.  Therefore,  the 
effect  of  smoothing  is  strongest  in  the  shadow  areas. 

(NU)  The  above  results  show  that  there  exists 
a  certain  tradeoff.  Combining  two  receive  channels 
(amplitude  only)  of  the  same  image  preserves  the 
original  focusing  but  provides  less  smoothing  due 
to  a  possible  high  correlation  between  the  chan¬ 
nels  (depending  on  the  clutter  type).  Combining 
35  GHz  and  94Ghz,  on  the  other  hand,  brings  to¬ 
gether  more  independent  information,  but  with  the 
possibility  to  deteriorate  the  focusing  quality.  This 
topic  will  be  analysed  further  in  the  next  para¬ 
graph. 

(NU)  The  target/background  contrast  can  be  as¬ 
sessed  by  looking  at  range  profiles  across  target 
positions.  Fig. 4a  shows  one  example  with  the  tar¬ 
get  located  at  a  range  of  about  210  meters.  At  first 
sight,  the  result  does  not  look  very  promising,  al¬ 
though  the  35-f-94GHz  profile  (solid  line)  shows  less 
variation  in  the  clutter  region  in  front  of  and  be¬ 
hind  the  target.  The  situation  looks  more  favor¬ 
able  if  one  considers  a  CFAR-level  profile  instead 
of  a  mere  range  profile.  As  a  simple  example,  the 
CFAR  level  ‘x’  for  each  pixel  value  ‘s’  is  calcu¬ 
lated  as  x  =  where  u,  and  a  are  the  mean  and 

cr  r 

standard  deviation,  respectively,  of  the  reflectivity 
across  the  entire  range  profile  with  the  10  strongest 
values  excluded  in  order  to  prevent  other  close-by 
targets  from  biasing  the  reference.  This  CFAR  level 
makes  use  of  the  reduced  standard  deviation  due  to 
smoothing  and  yields  an  improvement  in  contrast 
(fig. 4b). 

4.  Discrimination  algorithm 

performance 

(NU)  The  radar  based  discrimination  of  man¬ 
made  structures  embedded  in  a  natural  clutter 
background  consists  of  several  steps.  First,  the 
radar  image  layout  has  to  be  created  by  determin¬ 
ing  the  pixel  (resolution  cell)  size  and  performing 
the  projection  to  a  ground  map.  Next,  the  desired 
information  contents  of  each  resolution  cell  has  to 
be  chosen  and  the  preprocessing  done  accordingly 
(([2])  and  Novak  et  al.  ([4],  [6],  [7]))-  The  third 


step  would  be  a  prescreener  ([10],  [5],  [8],  [9]),  i.  e. 
some  sort  of  a  thresholding  procedure  which  nar¬ 
rows  down  the  number  of  possible  candidate  target 
sites  and  thus  reduces  the  computational  burden 
on  the  subsequent  processing  stages.  Based  upon 
the  output  of  the  prescreener,  feature  vectors  are 
then  constructed  that  help  to  discriminate  between 
man-made  targets  and  natural  background. 

(NU)  For  the  prescreener,  only  single  range  pro¬ 
files  or  a  small  number  of  range  profiles  must  be 
used.  The  discriminator,  too,  should  act  on  small 
areas  that  are  adapted  to  the  size  of  the  target 
that  one  wants  to  find.  This  takes  into  account 
the  fact,  that  in  realistic  systems,  the  incoming 
radar  backscatter  signals  form  a  time  sequence. 
That  means  that  each  detection  algorithm  has  to 
start  its  work  while  the  radar  image  is  still  being 
built  up.  The  final  product  of  the  overall  radar  pro¬ 
cessing  is  an  image  that  contains  within  each  pixel 
the  decision  result  “target  yes/no”.  This  image  can 
subsequently  be  used,  probably  in  connection  with 
an  original  radar  amplitude  image,  for  further  im¬ 
age  processing  techniques  like  pattern  matching, 
comparison  with  photographical  or  IR  images  etc. 

([3], [16])- 

4.1  Prescreener 

(NU)  The  prescreener  used  for  this  analysis  might 
be  called  a  shadow  filter  (SF).  It  acts  on  total  power 
HRR  profiles.  The  test  variable  T(n)  for  the  n-th 
pixel  within  a  HRR  line  is  computed  as: 

n  n+5 

T(n)  =  ^2  (Pi-P)~  0*  -  P) 

i=n  —  4  i=n-|-l 

(NU)  Here  p  =  total  power  average  over  one  HRR 
line  (500  to  600  pixels).  The  first  term  is  positiv, 
if  strong  scatterers  are  present.  The  second  term 
is  negativ,  if  a  shadow  is  present.  Therefore,  T(n) 
has  a  maximum  for  a  strongly  reflecting  structure 
casting  a  shadow.  The  SF  is  adapted  to  the  ex¬ 
pected  electrical  target  size  of  5  m.  T(n)  has  to  ful¬ 
fill  the  threshold  condition  T(n)>0.3m2  which  was 
determined  empirically.  Of  course,  this  SF  does  not 
only  show  small  structures  of  the  desired  size.  But 
if  there  is  a  large  structure  casting  a  shadow,  then 
only  its  trailing  edge  passes  the  prescreener  which 
may  reduce  considerably  the  amount  of  pixels  that 
are  left  for  subsequent  discrimination. 

4.2  Discrimination 

(NU)  For  discrimination  between  man-made  struc¬ 
tures  and  natural  clutter  background,  two  features 
were  used  that  were  analyzed  earlier  ([10]).  Only 
those  cells  that  had  passed  the  prescreener  were 
subjected  to  the  feature  test.  The  features  are: 
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Fi—  slope  of  straight  line  fit  to  total  power 
(8  strongest  cells  arranged  in  decreasing 
order) 

F2=  -£ean  ( °dd  +  eVen  Power) 


The  features  are  computed  over  areas  of  7  by  7 
pixels  centered  at  the  cell  under  test. 

(NU)  For  the  feature  test,  one  needs  references 
for  clutter  and  for  vehicles,  respectively.  The  clut¬ 
ter  reference,  consisting  of  1332  pixels,  was  a  mixed 
scene  containing  grass,  scattered  bushes  and  trees 
with  their  shadows,  taken  from  former  measure¬ 
ments  in  a  different  region  ([10]).  Thus,  one  avoids 
the  use  of  one  and  the  same  scene  for  training  and 
for  testing. 

(NU)  For  the  target  reference,  data  were  taken 
from  tower/ turntable  measurements  performed 
with  a  similar  radar  (CORA  94)  under  the  same 
depression  angle.  These  are  described  in  more  de¬ 
tail  in  [11]  and  [12].  The  data  belong  to  a  main 
battle  tank  covered  by  a  camouflage  net.  This  net 
reduced  the  overall  cross  section  by  less  than  1  dB, 
but  changed  the  geometry  and  signature  of  the  scat¬ 
tering  centers.  This  target  reference  is  by  no  means 
optimized  with  respect  to  the  specific  relocatable 
targets  that  are  in  the  scene.  Rather  it  serves  as  a 
generic  example  for  a  “man-made  object”  reference 
for  discrimination  against  a  natural  background. 

(NU)  The  references  are  stored  in  the  form  of 
mean  value  and  standard  deviation  of  the  respec¬ 
tive  feature,  based  on  the  feature  distributions  for 
the  reference  templates. 

(NU)  Next,  one  computes  as  described  in  [11] 
and  [10]  the  distances  of  the  feature  vector  under 
test  to  the  target  reference  and  to  the  clutter  refer¬ 
ence.  These  are  the  normalized  Euclidian  distances 
in  feature  space: 


Dt  — 


Dc  = 


(Fi  -  Ft,)2 


'E 


°Ti 


'E 


(Fi  -  Fa)2 


'a 


Dt  <  7  ■  Dc 

is  fulfilled  where  the  factor  7  <  1  has  to  be  chosen 
so  as  to  minimize  the  number  of  false  alarms  while 
not  affecting  the  detection  of  the  real  target.  Here, 
7=1  was  used  throughout. 


Fig.  5.  Detection  and  discrimination  probabilities  for 
targets  and  clutter  as  a  function  of  misregistration  be¬ 
tween  35  GHz  and  94  GHz  images 


5.  Results 

(NU)  The  test  was  performed  on  the  original 
35  GHz  and  94  GHz  scene  and  for  comparison  on 
the  combined  35+94  GHz  scene  with  optimal  corre¬ 
lation.  Afterwards,  the  two  images  were  offset  with 
respect  to  each  other  by  up  to  6  pixels  in  the  flight 
direction  whereas  in  range  the  relative  position  was 
unchanged.  This  was  done  to  study  the  influence  of 
mis-registration  on  algorithm  performance.  For  a 
quantitative  assessment  boxes  were  defined  around 
each  of  the  8  targets.  Within  each  box  the  num¬ 
ber  of  pixels  that  passed  the  prescreener  (NDET) 
were  divided  by  the  total  number  of  pixels  in  this 
box  (NPIX)  to  yield  the  detection  probability  P<fet. 
Next,  the  discrimination  probability  is  defined  as 
P<iiscr=NDIS/NDET,  where  NDIS  is  the  number 
of  pixels  that  passed  the  prescreener  and  the  fea¬ 
ture  test.  Table  2  summarizes  the  P^e,.  values. 


where  F^i  and  F ci  are  the  mean  values  and  aTi 
and  aci  are  the  standard  deviations  of  feature  F^ 
from  the  target  and  clutter  reference  distributions. 
The  use  of  the  above  distance  measure  in  feature 
space  is  justified  when  there  is  no  correlation  be¬ 
tween  the  features.  This  was  verified  in  the  present 
case. 

(NU)  Now,  any  pixel  under  test  is  said  to  belong 
to  the  class  of  “relocatable  target”  if  the  condition 


(NU)  As  can  be  seen  from  this  table,  the  normal 
case  seems  to  be  that  P diner  for  35+94  GHz  lies  be¬ 
tween  the  values  for  the  single  frequencies  (targets 
1,  3,  4,  6  and  7).  Vdiscr  is  lower  than  before  in  the 
case  of  targets  2  and  5  (where  #5  even  was  lost 
completely),  and  only  in  one  case  (target  8)  Pdiscr 
is  higher  than  before. 

(NU)  If  one  looks  at  the  effect  of  mis-registration, 
the  behaviour  is  not  uniform  either.  For  half  of 
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pixel  offset 

Tgt 

35  GHz 

94  GHz 

35+94  GHz 

1 

2 

3 

4 

5 

6 

1 

82.9 

62.1 

78.8 

44.4 

93.8 

46.7 

4.8 

- 

- 

2 

80.0 

70.0 

64.4 

64.0 

34.6 

2.0 

- 

- 

- 

3 

38.1 

91.8 

65.4 

28.6 

- 

2.0 

- 

- 

- 

4 

43.8 

84.1 

45.3 

56.9 

57.9 

66.1 

43.9 

15.2 

34.4 

5 

7.0 

41.1 

- 

18.0 

- 

7.4 

- 

1.5 

- 

6 

49.0 

73.0 

60.0 

44.9 

75.0 

53.2 

59.7 

75.0 

77.8 

7 

72.3 

49.0 

60.5 

50.4 

61.4 

44.6 

65.9 

56.3 

54.1 

8 

76.7 

66.0 

84.8 

53.1 

5.9 

- 

69.0 

9.4 

- 

ETgts 

8 

8 

7 

8 

6 

7 

5 

5 

3 

Table  2:  Discrimination  Probabilities  (in  %) 


the  targets  (1,2, 3, 4)  Pdiscr  decreases  obviously  as 
a  function  of  offset,  in  the  case  of  targets  1  and  4 
there  is  a  slight  maximum  around  an  offset  of  2  or  3. 
In  one  case  (target  5)  the  originally  lost  target  did 
reappear.  Targets  6  and  7  are  very  little  influenced, 
their  Pdiscr  remains  more  or  less  constant.  P discr 
of  target  8  shows  strong  variability  which  may  be 
due  to  ‘splitting’  (the  target  appears  as  two  peaks) 
when  the  offset  is  too  large. 

(NU)  If  one  combines  all  target  detections  and 
discriminations  (from  all  8  boxes)  and  likewise  all 
clutter  detections  and  discriminations  (from  out¬ 
side  the  boxes)  the  overall  behaviour  as  a  func¬ 
tion  of  mis-registration  can  be  studied  (fig. 5).  P det 
for  the  targets  increases  because  the  targets  are 
smeared  out  and  more  pixels  pass  the  prescreener. 
P det  for  the  clutter  decreases  slightly  because  due  to 
the  smoothing  effect  less  pixels  cross  the  threshold. 
Pdtacn  on  the  other  hand,  decreases  for  the  targets 
as  expected  and  as  seen  in  table  2.  P**cr  f°r  the 
clutter  remains  almost  constant.  It  must  be  kept  in 
mind,  though,  that  ‘clutter’  here  means  everything 
outside  the  target  boxes.  As  there  are  numerous 
buildings  in  the  area,  which  are  also  man-made  ob¬ 
jects,  they  are  discriminated  more  or  less  like  real 
targets  and  hence  constitute  true  false  targets,  and 
not  clutter  outliers. 

6.  Conclusions 

(NU)  The  present  preliminary  analysis  is  based  on 
multi-look  2-channel  amplitude-only  images.  It  has 
been  shown  that  the  correlation  between  35  GHz 
and  94  GHz  is  sufficient  to  warrant  a  precise  regis¬ 
tration  of  radar  maps.  On  the  other  hand  the  cor¬ 
relation  is  lower  than  between  the  two  orthogonal 
channels  of  a  single  frequency  and  therefore  offers 
more  independent  information. 

(NU)  This  becomes  visible  mainly  in  better 
smoothing  or  speckle  reduction  and  also  in  en¬ 
hancing  the  target/background  contrast  of  a  CFAR 
threshold  detector.  Concerning  a  discrimination  al¬ 
gorithm  which  consists  of  a  prescreener  and  a  fea¬ 


ture  test,  the  advantage  of  combining  two  frequen¬ 
cies  is  not  clearly  evident  in  the  present  example. 
This  may  be  due  to  the  statistical,  and  not  po- 
larimetric,  character  of  the  two  features  that  were 
used.  These  features  were  chosen  because  former 
experience  has  shown  ([10])  that  the  available  res¬ 
olution  of  only  1  meter  in  range  and  cross-range  is 
not  sufficient  for  the  optimum  use  of  polarimetry. 

(NU)  The  algorithm  performance  was  studied  as 
a  function  of  mis-registration  between  the  images 
of  the  two  frequencies.  An  offset  of  one  to  two  pix¬ 
els  is  not  necessarily  critical  as  some  of  the  targets 
show,  but  for  larger  shifts  there  is  a  rapid  decrease 
of  discrimination  probability.  On  the  other  hand, 
there  exist  cases  where  Pd,Jcr  is  almost  indepen¬ 
dent  of  mis-registrations  up  to  6  pixels.  Obviously 
they  consist  of  so  many  strong  pixels  that  an  addi¬ 
tional  smearing  does  not  change  the  statistics  that 
form  the  basis  for  the  two  features. 

(NU)  The  analysis  shows  that  a  simple  com¬ 
bination  of  amplitude  only  images  does  not  fully 
demonstrate  any  advantages  of  combining  35  GHz 
and  94  GHz.  In  a  next  step,  therefore,  the  images 
will  be  processed  with  enhanced  Doppler  resolution 
(the  range  resolution  is  limited  due  to  limited  band¬ 
width)  and  single-look  with  the  full  available  polari- 
metric  information  (scattering  matrix  at  35  GHz, 
Stokes  vector  at  94  GHz).  The  combined  image  will 
then  consist  of  six  channels,  and  the  discrimination 
features  have  to  be  constructed  accordingly. 
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PAPER  No:  8 

DISCUSSOR’S  NAME:  U.  Lammers 

COMMENT/QUESTION: 

You  mentioned  that  the  35  and  94  GHz  signals  are  generated  from  a  common  reference.  Was 
this  done  to  look  at  differential  phase  information,  and  did  you  actually  do  this? 


AUTHOR/PRESENTER’S  REPLY: 

Phase  information  is  included  in  the  polarimetric  description  (e.g.  Stokes’  parameters)  of  each 
return  signal.  These,  of  course,  will  be  compared  between  35GHz  and  94GHz  .  However,  this 
requires  an  absolutely  precise  (at  least  locally)  registration  of  the  two  images. 
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Introduction 

La  variete  des  demiers  conflits  ou  furent  engagees  les 
forces  alliees  a  montre  les  limitations  des  systemes 
d’armes  Air-Sol  existants  (fortes  contraintes  d’emploi, 
armements  trap  specifiques,  portees  de  tir  insuffisantes 
face  aux  defenses  Sol-Air,...). 

La  France  a  de  ce  fait  Fobjectif  de  se  doter  d’un 
armement  tactique  nouveau,  complementaire  des 
missiles  de  croisiere  du  type  SCALP. 

Cet  armement  de  conception  modulaire  designe  AASM 
(Armement  Air-Sol  Modulaire),  aura  des  portees 
superieures  a  15  km  et  devrait  etre  capable  de  deux 
classes  de  precision  metrique  et  decametrique. 

Avant  le  tir  des  armes,  il  est  necessaire  de  realiser  un 
certain  nombre  d’operations  telles  que  la  detection,  la 
reconnaissance  et  la  localisation  des  cibles  a  traiter. 

Le  RAFALE  va  disposer  de  capacites  nouvelles  pour 
satisfaire  ces  fonctions  grace  aux  modes  d’imagerie 
haute  resolution  de  ses  senseurs  internes  radar  et 
optronique. 

La  DGA  a  confie  a  TIIOMSON-CSF,  qui  est 
responsable  du  developpement  de  ces  modes,  F  etude: 

-  des  possibility  du  systeme  multi-senseurs  radar  et 
optronique, 

-  du  concept  d’emploi  du  systeme  multi-senseurs, 

-  des  traitements  necessaires  a  l’exploitation  temps 
reel  visuelle  ou  assistee  des  images. 

Ce  travail  s’appuie  sur  des  simulations  et  des  donnees 
reelles. 

La  presente  communication  a  pour  objet  de  presenter  les 
resultats  de  l'etude  en  cours. 

Liste  des  abreviations 

AASM:  Armement  Air-Sol  Modulaire 

BDA:  Battle  Dammage  Assessment 

BFR:  Basse  Frequence  de  Repetition 

DDT:  Domaine  De  Tir 

DGA:  Delegation  Generate  pour  l'Annement 

DRI:  Detection  Reconnaissance  Identification 

EM:  ElectroMagnetique 

GPS:  Global  Positionning  System 

HFR:  Haute  Frequence  de  Repetition 

HR:  Haute  Resolution 

MFR:  Moyeime  Frequence  de  Repetition 

MTI:  Moving  Target  Integration 

PLC:  Progranmiation  Logique  sous  Contraintes 

SACP:  Sol-Air  Courte  Portee 

SAR:  Synthetic  Aperture  Radar 

SER:  Signature  Equivalente  Radar 

TBA:  Tres  Basse  Altitude 

THR:  Tres  Haute  Resolution 


1-  Aspect  operationnel 

1-1  L' analyse  technique  du  besoin  operationnel 

Le  nouvel  armement  tactique  AASM  doit  permettre 
d’engager  un  objectif  defendu  par  des  missiles  Sol-Air 
en  restant  hors  de  leurs  domaines  de  tir,  la  portee  de 
l’arme  etant  de  15  km  a  tres  basse  altitude  et  plus  de  50 
km  a  haute  altitude. 

Parmi  les  autres  exigences  specifiees  pour  cet  armement 
on  peut  citer: 

-  tir  de  jour  et  de  nuit 

-  traitement  d’une  grande  variete  de  cibles:  fixes 
(batiments,  voies  de  communication,...),  depla5ables 
(batterie  S/A,  bateau  a  quai,...)  ou  mobiles 
(vehicule,  bateau  en  mer,...) 

-  capacite  multicible 

-  tir  toutes  altitudes 

-  capacite  "tire  et  oublie". 

Cet  armement  de  conception  modulaire  devra  etre 
capable  de  deux  classes  de  precision:  metrique  et 
decametrique. 

Ces  exigences  de  precision  necessitent  de  disposer  en 
amont  de  senseurs  capables  de  detecter,  de 
reconnaitre  et  de  localiser  precisement  les  cibles  a 
traiter. 

Ces  operations  peuvent  etre  realisees  avant  la  mission  a 
partir  des  informations  foumies  par  les  moyens  de 
reconnaissance  satellitaires  ou  aeroportes  ou  pendant  la 
mission  grace  aux  senseurs  de  bord.  Dans  la  plupart  des 
cas  les  deux  segments  contribuent  a  la  reussite  de  la 
mission. 

Les  fonctions  techniques  demandees  aux  senseurs  sont 
done: 

-  la  detection, 

-  la  reconnaissance, 

-  la  localisation. 

Les  besoins  de  detection,  reconnaissance  et  localisation 
pendant  la  mission  sont  differents  selon  Panne  et  la 
cible  consideres. 

Pour  un  armement  de  precision  decametrique  que  Pon 
supposera  &  guidage  inertie/GPS: 

-  le  besoin  de  detection/reconnaissance  est: 

-  faible  pour  les  cibles  fixes,  une  reconnaissance 
de  leur  contexte  est  a  priori  suffisante, 

-  fort  pour  les  cibles  deplacables  pour  lesquelles  il 
est  necessaire  de  continuer  la  presence,  mais 
dans  certains  cas  le  contexte  peut  etre  une  aide 
precieuse  (par  exemple  les  avions  sont  en 
general  sur  les  aires  de  stationnement  d’uue 
base  aerienne). 

-  le  besoin  de  localisation  precise  est: 

-  fort  pour  une  cible  fixe  dont  on  lie  coimait  pas 
assez  precisement  les  coordonnees  absolues  a 
Fissue  de  la  preparation  de  mission. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar",  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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-  fort  pour  une  cible  deplafable  dont  la 
prelocalisation  sera  dans  la  plupart  des  cas 
imprecise 

( moyen  d’ecoute  EM,  par  exemple,  pour  la  localisation 
de  radar  sol) 

Pour  un  armement  dc  precision  metrique  qu’on 
supposera  muni  d’un  senseur  de  guidage  a  imagerie: 

-  le  besoin  de  detection/reconnaissance  est: 

-  fort  on  faible,  pour  les  cibles  fixes,  selon  le 
concept  d’armement  considere  (controle  actif  ou 
pas  avant  le  tir), 

-  fort  pour  les  cibles  depla5ables  pour  lesquelles  il 
est  necessaire  de  confirmer  la  presence, 

-  fort  pour  les  cibles  mobiles, 

-  le  besoin  de  localisation  precise  est: 

-  faible  ou  moyen  selon  la  qualite  de  la 
prelocalisation,  la  nature  de  la  cible  et  le 
concept  d’annement. 

Ces  considerations  sur  les  besoins  techniques  montrent 
que  l’analyse  doit  etre  menee  au  cas  par  cas.  Ceci  nous 
a  conduit  a  considerer  un  ensemble  de  scenarios 
caracteristique  des  conditions  d’emploi  du  systeme 
multi-senseurs  pour  analyser  le  besoin  teclmique  et  les 
performances  du  systeme  multi-senseurs. 

1-2  La  definition  de  scenarios  de  reference 

La  definition  des  scenarios  de  reference  a  ete  realisee  en 
trois  etapes: 

-  analyse  des  cibles  d'interet  et  choix  de  cibles  de 
reference, 

-  positionnement  de  ces  cibles  dans  des  scenarios 
pennettant  de  definir  les  conditions  d'emploi  du 
systeme. 


-  renseignement  des  caracteristiques  des  cibles,  des 
fonds,  des  conditions  meteorologiques  et  climatiques 
pour  le  calcul  des  performances  du  systeme  multi- 
senseurs. 

Etant  donne  le  grand  nonibre  de  cibles  d'interet  (>100), 
des  criteres  de  classement  ont  ete  definis: 

-  le  type  de  cibles:  Fixe/Deplafable/Mobile, 

-  la  durete:  Forte/Faible, 

-  la  forme:  Etendue/Ponctuelle 

Nous  avons  ensuite  selectionne  des  cibles  type  dans 
chacune  des  classes  en  fonction: 

-  du  cadre  de  l'etudc,  limite  aux  cibles  fixes  et 
deplacables, 

-  du  nombre  de  cibles  de  la  classe, 

-  de  la  priorite  operationnelle. 

Les  cibles  selectionnees  ont  ensuite  ete  placees  dans  des 
scenarios. 

Nous  avons  considere,  pour  chaque  scenario,  un  certain 
nombre  d'elements  dimensioimants  utiles  pour  le  calcul 
des  performances  des  senseurs  et  l'analyse  de  leur 
emploi  dans  la  phase  d'attaque. 

Ces  elements  dimensioimants  sont: 

-  les  defenses  Sol-Air, 

-  le  caractere  monocible  ou  multicible  de  l'objectif, 

-  le  type  de  mission, 

-  le  profil  d'attaque  (BA:  200  ft,  MA:  20000  ft,  HA: 
40000  ft), 

-  le  profil  de  terrain, 

-  le  type  de  milieu, 

-  le  clirnat. 

Les  catechistiques  principales  des  cibles  et  des 
scenarios  retenus  sont  presentees  dans  la  planche  ci- 
apres. 


SYNTHESE  DES  OBJECTIFS  ET  DE  L'ENVIRONNEMENT  SOL-AIR  DES  SCENARIOS 


SYNTHESE /  SCENARIOS  PK #>#S0S 


Objectif 

Mobility 

Caracteristiques 

/Durete  /  Environnement 

Defenses  sol-  air 

Profil 

1-1  Pont/  Fleuve 

Fixe 

/  Forte  /  Urbain 

SACP 

BA-MA-H/ 

1-2 

Pont suspendu 

Fixe 

/  Forte  /  Rural 

Non  defendu 

BA-MA-HA 

2-1  Bateau  a  quai 

Semi-mobile  /  Forte  /  Urbain 

SAMP  +  SACP 

HA 

. - 

12-2  £ 

lateau  en  mer 

Semi-mobile  /Forte  /  Marin 

SACP  +  SATCP 

MA- BA 

3-2 

Piste 

Fixe 

/  Faible  /  Dbsertique 

SACP  +  SATCP 

MA-HA 

B 

Radar  tactique 

Fixe 

/  Faible  /  Vegetation 

my~s< 

£ 

Hangarettes 

Fixe 

/  Forte  /  sable  &  beton 

Soutes 

Fixe 

/  Faible  /  sable  &  beton 

tm  a 
;  e: 

Avion 

Semi-mobile  /Faible  /Bbton 

md : 

-  t 
...  £ 

N 

N 

VGhicule  S/A 

Semi-mobile  /Faible  /  Terre 

_L  .  ,^T 

Shelter 

Semi-mobile  /  Faible  /  Sable 

OyL  _ 

E 

Bunker 

Fixe 

/  Forte  /  Sable  &  beton 

1-3  Les  etapes  d’une  attaque  air-sol 

Nous  terminons  l’analyse  de  1’aspect  operationnel  par  une  presentation  des  differentes  etapes  d’une  attaque  au  sol. 
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DIFFERENTES  PHASES  D'UNE  ATTAQUE  AIR-SOL 


La  mise  en  oeuvre  des  senseurs  pour  la  detection,  la  reconnaissance,  et  la  localisation  s’effectue  dans  la  phase  approche. 


2-  Les  senseurs 

Le  RAFALE  dispose  de  deux  senseurs  de  bord 
susceptibles  de  lui  foumir  des  images: 

-  le  radar  RBE2  (Radar  a  Balayage  Electronique  2 
plans) 

-  l’optronique  OSF  (Optronique  Secteur  Frontal). 


Le  RBE2  et  l’OSF  sont  les  deux  senseurs  multifonctions 
du  RAFALE  couvrant  le  secteur  frontal  de  F  avion.  Ils 
sont  tous  les  deux  localises  dans  la  pointe  avant  de 
l’avion  comxne  le  montre  F  image  ci-dessous. 
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Le  RBE2  est  un  radar  en  bande  X  a  balayage 
electronique  deux  plans,  multi-formes  d’onde  (BFR, 
MFR  et  HFR)  qui  disposent  de  fonctions: 

-  air-air  d’ interception  et  de  combat:  recherche  et 
pistage  multicible,  acquisition  automatique  et 
multipoursuite  en  combat, 

-  TBA  foumissant  une  carte  temps  reel  du  sol  pour  le 
Suivi  De  Terrain  tout  temps  de  1’ avion, 

-  air-sol:  cartographie  faisceau  reel  et  imageries  SAR. 

-  air-mer:  detection  et  poursuite  de  cibles  marines. 

Le  RBE2  dispose,  dans  sa  fonction  Air-Sol,  de  deux 
modes  d’imagerie  SAR: 

-  un  mode  HR  de  resolution  decametrique  foumissant 
l’image  d’une  zone  de  terrain  pouvant  atteindre 
quelques  km  de  cote, 

-  un  mode  THR  de  resolution  metrique  couvrant  une 
zone  plus  petite. 

L’OSF  est  un  equipement  multi-fonctions  assurant: 

-  en  air-air  interception  et  combat:  recherche, 
detection  et  multipistage  2D,  mono-poursuite  3D  et 
identification 

-  en  air-sol:  mono-poursuite  3D  et  identification 

-  en  air-mer:  detection  et  multi-pistage  2D,  mono- 
poursuite  3D  et  identification 

L’OSF  disposera  de  deux  modes  d’imagerie: 

-  un  mode  visible  avec  une  resolution  de  quelques 
dizaines  de  prad, 

-  un  mode  IR  avec  une  resolution  moins  bonne. 

3-  Les  possibilites  du  systeme  multi-senseurs 

Pour  juger  des  possibilites  du  systeme  multi-senseurs 
des  criteres  devaluation  ont  ete  definis.  Ces  criteres 
sont  de  deux  types: 

-  caracteristiques  (parametres  qui  lie  dependent,  au 
premier  ordre,  que  du  mode  senseur  considere,  par 
exemple  la  resolution  oil  le  domaine  angulaire 
accessible), 

-  perfomiances  (parametres  qui  dependent  du  mode  et 
du  scenario  considere,  par  exemple  la  portee  de 
reconnaissance). 

Les  definitions  de  ces  criteres  ont  ete  harmonisees  afin 
d’etre  en  mesure  de  comparer,  sans  ambigui'te,  les 
resultats  (definition  unique  et  memes  hypotheses  de 
calcul  pour  les  deux  senseurs). 

Les  perfomiances  du  systeme  multi-senseurs  sont 
ensuite  estimees  a  l’aide  des  scenarios  de  reference 
presentes  dans  le  §1. 

3-1-  Listes  des  criteres  retenus 

Les  criteres  devaluation  retenus  se  repartissent  en  deux 
groupes: 

-  les  criteres  quantitatifs 

-  portees  de  detection, 

-  portee  de  reconnaissance, 

-  portee  ^identification, 

-  precision  de  localisation  absolue, 

-  precision  de  localisation  relative, 

-  taille  de  la  zone  imagee 

-  temps  d'obtention  des  informations, 

-  cadence  de  rafraichissement, 


-  domaine  angulaire  accessible, 

-  les  criteres  qualitatifs 

-  sensibilites  aux  conditions  climatiques  et 
meteorologiques, 

-  utilisation  jour/nuit, 

-  discretion, 

-  dependance  vis  a  vis  de  la  preparation  de 
mission, 

-  capacite  devaluation  du  resultat  de  tir, 

-  robustesse. 

3-2-  Les  caracteristiques  images 

Les  parametres  retenus  pour  caracteriser  les  differents 
modes  d'imagerie  sont  les  suivants: 

-  dimension  (2D  ou  3D)  et  parametres  primaires 
correspondants  (distance,  gisement,  site), 

-  resolution  spatiale, 

-  taille  de  la  zone  imagee 

-  temps  d'obtention  des  informations, 

-  cadence  de  rafraichissement, 

-  domaine  distance, 

-  domaine  angulaire  accessible, 

-  domaine  d'emploi  porteur. 

Afin  de  pouvoir  comparer  directement  les 
caracteristiques,  les  parametres  qui  decrivent 
spatialement  l'image  (taille  et  resolution)  sont 
presentees  dans  le  repere  normal  terrestre  associe  a 
l’avion  en  coordomiees  spheriques  et  cartesiennes.  Ces 
caracteristiques  sont  illustrees  par  des  cas  typiques  de 
visee  a  des  distances  donnees  differentes. 

3-3-  Les  performances 

3-3-1  calcul  des  perfomiances  de  DRI  (Detection, 
Reconnaissance,  Identification). 

A-  Evaluation  des  portees  de  DRI  en  optronique 

La  methode  devaluation  des  perfomiances  utilise  les 
criteres  dits  «  de  Johnson  ».. 

La  cible  est  modelisee  par  un  rectangle  de  surface 
equivalente  a  sa  surface,  dans  lequel  on  place  une  mire 
creneau  dont  la  periode  depend  du  critere  considere: 

-  1  cycle  pour  la  detection 

-  3,5  cycles  pour  la  reconnaissance 

-  7  cycles  pour  l’identification 


Modele  frequentiel  de  la  cible  sur  le  fond 
Cas  de  la  reconnaissance 
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Le  contraste  de  la  mire  s’exprime  en  temperature  pour 
l’infrarouge  et  en  albedo  pour  le  visible.  Ce  contraste 
est  ensuite  calcule  apres  transmission  dans  Fatmosphere 
consideree  (code  LOWTRAN).  On  en  deduit  ainsi  le 
contraste  apparent,  fonction  de  la  distance 
d’ observation. 

L’equipement  est  modelise  par  une  courbe  traduisant 
son  aptitude  a  discriminer  («  resoudre  »)  un  ecart  de 
temperature  (cas  de  Finfrarouge)  ou  d’albedo  (cas  du 
visible)  pour  une  frequence  spatiale  donnee.  Cette 
demiere  s’exprime  directement  en  divisant  la  periode  de 
la  mire  par  la  distance  d’ observation. 

Cette  discrimination  est  proportiomielle  au  bruit  do 
detection  et  inversement  proportioimelle  a  la  fonction  de 
transfert  de  modulation  (FTM).  Cette  demiere  prend  en 
compte  la  geometrie  du  detecteur,  la  qualite  optique,  le 
traitement  electronique  ainsi  que  la  visualisation  et  les 
conditions  d’observation  par  l’operateur. 

L’intersection  de  ces  2  courbes  correspond  a  une 
distance  pour  laquelle  la  frequence  spatiale  pennet  de 
rester  au-dessus  du  seuil  en  temperature  ou  en  albedo: 
e’est  la  portee  recherchee. 


Contraste  Cible/Fond 


Cette  metliode  a  ete  eprouvee  depuis  2  decennies  par  de 
tres  nombreux  essais  sur  des  cibles  reelles  et  avec  des 
equipements  optroniques  differant  par  leur  bande 
spectrale  et  leur  champ  d’observation. 

Cette  metliode  de  calcul  des  portees  de  DRI  est 
particulierement  efficace  dans  la  comparaison 
d’equipements  ou  lorsque  l’on  fait  varier  des  parametres 
de  nature  teclmique.  Elle  doit  ensuite  etre  ponderee  par 
le  contexte  operatiomiel  et  le  type  de  cible. 

On  ne  traitera  pas  de  la  nieme  maniere  la 
reconnaissance  d’une  base  aeriemie  et  celle  d’une 
batterie  sol/air.  Dans  le  cas  d'un  objectif  etendu  conmie 
une  base  aerienne,  la  detection  d’un  element 
caracteristique  (la  piste  par  exemple)  domiera  acces 
instantanement  a  I'identification  de  l'objectif.  Dans  le 
cas  d’une  cible  depla?able  dont  le  type  est  parfaitement 
determine  (avion  de  combat  par  exemple)  la 
reconnaissance  sera  effective,  dans  certains  cas  ,  des  sa 
detection. 

On  interpretera  encore  differenmient  un  calcul  dans  le 
cas  d’une  mission  preparee  et  dans  le  cas  de  l’attaque 
d’unc  cible  d’opportunite.  Dans  le  cas  d’une  mission 
preparee,  on  detectera  souvent  une  cible  fixe  par  la 
reconnaissance  du  contexte  enviromiant. 

Chaque  cible  pent  done  relever  d’une  definition 
particuliere  conditionnee  a  la  fois  par  sa  nature  propre  et 
le  type  de  mission. 


Dans  le  domaine  de  la  reconnaissance  aeroportee,  un 
ST.ANAG  (le  3769)  s’attache  a  definir,  pour  chaque 
cible,  ses  dimensions  ainsi  que  la  dimension  du  detail 
pennettant  d’acceder  a  la  detection,  puis  a  la 

reconnaissance,  etc . Une  analyse  au  cas  par  cas  de  ce 

STANAG,  qui  a  ete  defini  a  Forigine  pour  les 
equipements  dotes  de  film  photographique  (et  done 
operant  dans  le  domaine  visible),  se  recoupe  tres  bien 
avec  les  criteres  de  Johnson. 


B-  Evaluation  des  portees  de  DRI  en  radar 

La  methode  de  prediction  de  performances  utilisee 
s'appuie  sur  les  travaux  realises  par  THOMSON-CSF 
dans  le  domaine  des  SAR  satellitaires. 

Le  principe  general  de  cette  metliode  est  de  considerer 
que  la  distribution  du  signal  dans  l'image  de  la  cible  et 
des  fonds  est  de  Rayleigh  et  que  la  retrodiffusion  des 
cibles  est  principalement  fonction  de  leur  orientation. 

On  considere  que  la  performance  obtenue  par  un 
operateur  humain  entraine  est  identique  a  celle  d'un 
detecteur: 

-  simple  seuil  lorsque  l'operateur  ne  prend  pas  en 
compte  les  details  de  l'image  de  la  cible  mais 
effectue  uniquement  une  analyse  energetique, 

-  double  seuil  lorsque  l'operateur  analyse  le  contenu 
de  l'image  de  la  cible  pour  apprecier  l'etendue  de  la 
tache  de  retrodiffusion. 

Lorsque  l'image  de  la  cible  s'etend  sur  plus  de  cent 
pixels  l'operateur  combine  une  analyse  energetique  et 
statistique  en  prefiltrant  l'image  pour  adapter  le  nombre 
de  pixels  a  ses  capacites  d'analyse.  Dans  ce  cas  la 
distribution  des  macro-pixels  obtenus  apres  filtrage  est 
proche  d'une  loi  nonnale. 

En  photo-interpretation,  on  modelise  la  performance  de 
visibilite  des  cibles  sur  un  terrain  homogene  par  une 
detection  double  seuil. 

On  etend  ce  principe  a  la  reconnaissance  et  a 
I'identification  en  considerant  que  ces  operations 
necessitent  la  visibilite  d'un  nombre  de  pixels  superieur. 
On  considere  que  les  performances  de  reconnaissance  et 
d'identification  d'une  cible  peuvent  etre  calculees  par  le 
produit: 

-  d'une  probabilite  de  reconnaissance  de  l'objet  isole 
dans  des  conditions  favorables  (fond  tres  faiblement 
retrodiffusant  et  bruit  thermique  tres  faible), 

-  d'une  probabilite  de  visibilite  d'un  nombre  minimal 
de  pixels  p  panni  n  composant  l'image  de  l'objet. 

Le  rapport  p/n  est  de  l'ordre  de  50%  pour  la  detection  de 
petites  cibles,  il  est  superieur  pour  reconnaitre  et 
identifier. 

Resultat  en  exploitation  visuelle  et  en  exploitation 
assistee 

En  exploitation  visuelle,  le  resultat  est  ajuste  en 
fonction  des  delais  d'analyse  consideres  et  de  la  capacite 
de  concentration  des  operateurs  concemes;  le  resultat 
final  etant  une  interpolation  entre  les  perfonnances 
simple  seuil  et  double  seuil. 

Dans  le  cas  d'une  exploitation  assistee  on  fait 
l'hypothese  que  le  traitement  foumit  un  resultat 
equivalent  a  celui  d'un  detecteur  double  seuil. 
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Etapcs  du  calcul  des  portees  de  DRI 

1-  Pd  et  Pfa  globale  souhaitee  sont  fixees  (meme 
hypothese  pour  le  radar  et  pour  l'optronique) 

2-  recherche  de  la  Pd  elementaire  (Pde)  fonction  de  la 
Pd  globale  et  du  rapport  p/n  (pixels  a  observer), 
(calcul  iteratif) 

n 

Pd  =  2  C[  •  Pde 1  (1  -  Pde)n-’ 

i=P 

3-  recherche  de  la  Pfa  elementaire  (Pfae)  fonction  de  la 
Pfa  globale  (calculee  par  simulation) 

4-  calcul  du  contraste  necessaire 
C  =  log(Pfae)  /  log  (Pde) 

5-  calcul  du  coefficient  de  retrodiffusion  equivalent  au 
bruit  radar  Neao 

C  =  (1+  (1-k)  CTo/Nerro)  •  (SER/S  +  Necro)  /  (oo+  Neoo) 
avec: 

-  SER:  surface  equivalente  radar  de  la  cible 

-  S:  surface  apparente 

-  k:  coefficient  de  ponderation  pour  prise  en 
compte  de  l'ombre  ( 0<k<l ) 

6-  calcul  de  la  portee  en  utilisant  l'equation  du  radar. 

La  suite  des  travaux,  avec  en  particulier  l'acquisition  de 
nouvelles  images  reelles  ou  simulees,  permettra 
d'ajuster  cette  methode  et  de  renseigner  les  parametres 
(valeurs  de  SER,  capacite  de  discrimination,...). 

3-3-2  Calcul  des  performances  de  localisation 

Le  calcul  des  performances  de  localisation  est  realise  a 
l'aide  d’une  methode  statistique  basee  sur  une 
decomposition  en  erreurs  primaires. 

Les  principales  erreurs  primaires  considerees  sont: 

-  la  localisation  porteur  (erreurs  en  position,  vitesse  et 
angle) 

-  P harmonisation  des  senseurs, 

-  les  erreurs  de  formation  de  P  image  dues  au  relief,  a 
la  direction  de  pointage,... 

-  les  erreurs  de  designation  de  la  cible  fonction  de  la 
resolution  de  mode. 

3-4-  Comnientaires  sur  les  caracteristiques  et 
performances 

Les  premiers  resultats  obtenus  permettent  de  degager  un 
certain  nombre  de  tendances  qui  devront  etre  confirmees 
par  la  suite  des  travaux. 

L’examen  des  caracteristiques  et  performances  des 
modes  montrent  de  nombreux  caracteres  de 
complementarite: 

1-  les  images  de  la  scene  foumies  par  radar  et 
optronique  correspondent  a  des  plans  differents: 

-  le  radar  donne  une  image  plan  horizontal:  distance- 
gisement 

-  1  optronique  une  image  orthogonale  a  Laxe 
d’observation,  done  dans  le  cas  des  sites  faibles  une 
image  plan  vertical:  gisement-site. 

La  fusion  des  deux  images  permet  de  reconstruire 
P  image  3D:  distance-gisement-site.  11  faut  noter 
toutefois  qu’il  existe  des  possibility  d’acquisition  de  la 
troisieme  dimension  pour  chaque  capteur: 


-  directement:  par  sitometrie,  dans  toute  P image,  pour 
le  radar,  par  telemetrie,  au  centre  de  P  image,  pour 
P  optronique. 

-  indirectement  par  P  utilisation  d’un  modele 

numerique  de  terrain. 

2-  l’optronique  foumit  une  image  continue  de  la  scene 
(rafraichie  a  haute  cadence),  le  radar  foumit  un 
ensemble  de  « cliches »  separes  au  minimum  par 
quelques  secondes. 

3-  la  variability  des  performances  de  DRI  est 
essentiellement  due: 

-  a  la  presentation  de  la  cible  en  radar,  quelques 
degres  d’ecart  et  P  image  de  la  cible  peut  etre  tres 
differente, 

-  a  la  distance  en  optronique,  celle  ci  joue  a  la  fois 
directement  sur  la  resolution  metrique  (effet  zoom) 
et  sur  l’attenuation  atmospherique  particulierement 
sensible  en  optronique. 

A  contrario,  les  performances  optroniques  sont  peu 
sensibles  a  la  presentation  de  la  cible  tout  cornrne  celles 
du  radar  a  la  distance. 

Pour  le  radar  il  est  possible  de  degager  les 
indications  suivantes: 

-  la  detection  et  la  reconnaissance  du  contexte,  qui 
utilisent  la  visibility  des  textures  de  terrain 
associees  a  des  macro-objets  (routes,  fleuves,  ...)  ou 
des  associations  de  points  brillants  (alignement  de 
pylones,  de  batiments,  •  ...),  sont  bonnes  car 
P  observability  des  elements  de  contexte  ne  depend 
pas,  au  premier  ordre,  de  la  presentation.  Le  mode 
SAR  HR  s’avere  dans  la  plupart  des  cas  suffisant 
pour  cette  operation. 

-  La  detection  et  la  reconnaissance  des  objets 
etendus,  qui  presentent  quelle  que  soit  la  direction 
d’observation  suffisamment  de  reflecteurs  pour 
former  une  image  enveloppe  exploitable,  sont 
possibles.  La  premiere  des  cibles  caracteristiques,  le 
pont  a  piles,  est  typique  du  phenomene.  L’ image  du 
pont  varie  avec  sa  presentation,  les  piles  signeront 
peu  dans  l’axe  du  pont,  mais  il  subsiste  toujours  des 
elements  de  structure  tel  que  la  rambarde  observable 
dans  toute  les  directions.  Avec  le  mode  SAR  HR,  il 
peut  exister  des  directions  d’observation  (au 
voisinage  de  l’axe  par  exemple)  ou  le  pont  est  peu 
observable,  en  SAR  THR  la  silhouette  devrait  etre 
toujours  visible. 

-  La  detection  et  la  reconnaissance  des  petites  cibles 
de  type  vehicules  ou  avions  de  combat  est  un 
probleme  complexe  en  cours  d’ analyse.  Le  mode 
SAR  HR  n’est  pas  adapte  a  ce  type  de  cibles  et  pour 
le  mode  SAR  THR  on  peut  indiquer  certaines 
tendances  qui  devront  etre  confirmer  par  des 
experimentations: 

-  la  detection  des  cibles  est  possible,  le  principal 
element  conditionnant  semble  etre  le  fond  sur  lequel 
se  trouve  la  cible:  on  voit  a  quelques  dizaines  de 
NM  un  avion  sur  une  piste,  on  ne  voit  pas  un 
vehicule  S/A  dans  de  Pherbe. 

-  La  reconnaissance  des  cibles  est  possible,  dans  la 
mesure  ou  l’on  peut  exploiter  le  parametre  le  plus 
discriminant:  les  dimensions.  Par  exemple,  il  est 
possible  de  discriminer  un  avion  de  combat  d’un 
avion  de  transport.  Il  sera  plus  difficile  de  le 
discriminer  avec  un  objet  de  dimensions  proches: 
par  exemple  un  camion  citeme.  Pour  cette 
discrimination  il  est  necessaire  d’utiliser  le  contenu 
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de  l’image.  Lorsqu’on  atteint  des  resolutions 
metriques  les  points  brillants  sont  resolus  mais  ne 
presentent  pas  un  rayonnement  isotrope.  Par 
consequent,  si  l’image  de  l'objet  est  suffisamment 
forcnee  pour  calculer  ses  dimensions,  elle  est  aussi 
Ires  fluctuante  en  fonction  de  sa  presentation. 

Avec  un  mode  MTI  il  est  possible  de  detecter  des  cibles 
mobiles  dans  1’ image  radar  si  celles-ci  ont  une  vitesse 
radiale  suffisante:  superieure  a  quelques  m/s. 

Pour  l’optronique,  dont  la  dependance  aux 
conditions  meteorologiques  est  bien  connue,  il  est 
possible  de  degager  les  indications  suivantes: 

-  la  detection  et  la  reconnaissance  du  contexte  et 
d’objets  etendus  sont  possibles  aussi  bien  en 
visible  qu’en  IR;  le  fait  d'imager  a  site  faible 
(S<10°)  favorisant  plutot  les  elements  contextuels 
ou  les  cibles  qui  presentent  une  extension  verticale 
importante. 

-  la  detection  ,  la  reconnaissance  et  l 'identification 
de  petits  objets  sont  possibles  aussi  bien  en  visible 
qu’en  IR. 

La  resolution  etant  angulaire,  les  portees  sont  tres 
variables  en  fonctions  des  cibles  auxquelles  on 
s’interesse.  Les  portees  sont  notablement  plus 
importantes  sur  une  base  aerienne  que  sur  une  batterie 
sol/air  par  exemple,  surtout  si  on  les  pondere  par  les 
elements  de  contexte  et  de  mission  cites  precedenunent. 

-  L’image  presentee  permet  une  exploitation  visuelle 
aisee.  Ceci  est  valable  dans  tous  les  domaines 
spectraux  relevant  de  l’optronique. 

Sur  toutes  les  cibles  concemees  par  une  arme  tactique, 
la  resolution  et  les  champs  consideres  offrent  un  niveau 
d’information  suffisant  pour  que  le  cerveau  humain 
fasse  son  travail  de  correlation  et  d’interpretation. 

E11  particulier,  l’exploitation  visuelle  de  1’ image  est  peu 
dependante  du  site  d’observation,  les  transformations 
geometriques  necessaires  etant  realisees  instinctivement 
par  l’operateur. 

-  Du  fait  de  la  contribution  de  la  transmission 
atmospherique,  les  portees  sont  beaucoup  plus 
importantes  a  haute  altitude  qu’a  basse  altitude, 
I’epaisseur  d’atmosphere  traversee  etant  plus  faible. 
Cette  constatation  s’applique  a  tout  equipement 
optronique,  et  en  particulier  a  ceux  dedies  a  la 
reconnaissance. 

-  Le  debattement  de  la  ligne  de  visee  et  la  frequence 
de  renouvellement  de  l’image  (frequence  video) 
autorisent  1’exploitation  de  1’ image  dans  des 
conditions  extremes  de  debattement  angulaire  et 
devolution  du  porteur,  grace  en  particulier  a  un 
tracking  operant  a  la  frequence  video. 

3-5  Conclusions  actuelles  sur  les  possibilites  du 
systeme  multi-senseurs 

Importance  du  contexte  pour  detecter  et  reconnaitre. 

Dans  beaucoup  de  cas  l’utilisation  du  contexte  est  une 
aide  precieuse  pour  detecter  et  reconnaitre: 

-  toujours  pour  les  cibles  fixes, 

-  souvent  pour  les  cibles  deplacables  (avions,  bateau  a 
quai,  convoi,  trains,...). 

Le  radar  offre  de  bomies  perfonnances  de  detection  et 
de  reconnaissance,  a  grande  distance,  d'objets  etendus 


tels  que  les  routes  ou  les  pistes  ,  et  a  fortiori  du 
contexte. 

Le  radar  detecte  a  longue  distance  des  objets  de  petite 
taille  comrne  des  avions  sur  une  aire  de  stationnement, 
il  peut  avec  son  mode  THR  discriminer  des  objets  de 
tailles  differentes,  par  exemple,  un  avion  de  combat 
d’un  avion  de  transport  militaire. 

L'optronique  offre  de  bonnes  performances  de 
reconnaissance  et  ^identification  sur  toutes  les  cibles  de 
petite  taille,  et  a  fortiori  sur  les  cibles  plus  etendues. 

Pour  la  localisation  des  cibles,  les  causes  d’erreur 
principales  sont  differentes  selon  les  capteurs  mais  dans 
tous  les  cas  on  constate  l'effet  distance:  l'erreur  de 
localisation  croit  lineairement  avec  la  distance.  Il  existe 
toutefois  un  cas  particulier  ou  l'erreur  n'est  pas 
proportiomielle  a  la  distance  du  point  designe,  c'est  celui 
oil  Ton  peut  utiliser  un  point  d’appui  de  coordonnees 
comiues  present  dans  l’image.  Une  double  designation 
point  d’appui  /  cible  permet  alors  de  s’affranchir  de 
l’effet  distance.  L'erreur  resultante  est  alors 
principalement  fonction  des  deformations  de  l'image. 

Le  travail  a  venir  permettra  de  preciser  les 
performances  attendues  du  systeme  en  exploitation 
visuelle  et  en  exploitation  assistee. 


4-  Les  traitements 

La  suite  des  travaux  permettra  de  defmir: 

-  le  concept  d’emploi  des  senseurs, 

-  les  assistances  necessaires  a  l’exploitation  des 
images. 

4-1  Le  concept  d’emploi  du  systeme 

Le  but  de  cette  tache  est  de  defmir,  a  partir  des 
possibilites  du  systeme  multi-senseurs,  les  conditions 
d’emploi  pemiettant  de  realiser  une  utilisation  optimale 
des  differents  modes  d’imagerie  radar  et  optronique 
ainsi  que  les  traitements  qui  permettront  d’effectuer 
cette  mise  en  oeuvre. 

Pour  ce  faire  on  utilisera  une  demarche  analytique 
s’appuyant  sur  les  scenarios  de  reference. 

La  mise  en  oeuvre  des  capteurs  est  conditionnee 
principalement  par: 

-  les  contraintes  techniques  d’emploi  consequences 
des  caracteristiques  des  modes  presentes  dans  le 
paragraphe  precedent, 

-  les  contraintes  operationnelles  a  respecter  :  necessity 
de  reconnaitre  la  cible  avant  tir,  affinage  de 
localisation  necessaire,  discretion,  temps  sur  zone  a 
respecter,  secteur  d’attaque,  distance  de  tir 
minimale ... 

-  le  domaine  de  perfonnances  des  capteurs 

Le  plein  ernploi  du  systeme  dans  l’environnement 
fortement  evolutif  du  combat  aerien  amenera 
vraisemblablement  a  defmir  des  logiques  de  pointage 
automatique  ou  de  recalage  des  modes  petit  champ  a 
partir  d’unc  designation  effectuee  dans  un  mode  grand 
champ,  a  priori  radar  sauf  contraintes  d’approche  tres 
basse  altitude  ou  de  discretion. 
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Vu  ie  grand  nombre  de  parametres  a  prendre  en  compte 
pour  realiser  une  mise  en  oeuvre  optimale  du  systeme,  il 
sera  surement  necessaire  de  defmir  des  fonctions 
permettant  d’aider  a  la  mise  en  oeuvre  du  systeme 
multi-senseurs  et  d’en  predire  les  performances  en 
fonction  des  utilisations  proposees.  Une  de  ces  fonctions 
realiserait  la  planification  des  ressources  en  preparation 
de  mission,  l’autre  proposerait  une  replanification  en  vol 
a  partir  des  evenements  imprevus  qui  ont  conduit  a  ne 
pas  respecter  la  preparation  de  mission.  De  telles 
fonctions  pourraient  etre  realisees  a  l’aide  des 
techniques  de  PLC  classiques  (en  preparation  de 
mission)  et  Anytimes  (en  vol),  les  algorithmes  de  PLC 
Anytimes  permettent  de  disposer  d'un  resultat  en 
controlant  le  temps  de  reponse.  La  PLC  deja  largement 
utilisee  pour  1 ’allocation  de  ressources  presente,  par 
rapport  a  la  programmation  logique,  l’avantage  de 
l’evolutivite. 

4-2  Les  assistances  a  I’exploitation  des  images 

Afm  d'alleger  la  charge  de  travail  du  pilote,  des 
traitements  bord  peuvent  l'assister  dans  les  fonctions  de 
DRI  et  de  localisation 

Les  fonctions  de  detection  et  de  reconnaissance  de 
contexte  et  d'objets  etendus  ont  fait  l’objet  de  travaux  a 
THOMSON-CSF  en  radar  conmre  en  optronique.  Elies 
sont  basees  sur  Putilisation  de  primitives  graphiques 
decrivant  des  elements  observables  du  contexte  (routes, 
batiments,  pylones,...).  Les  formes  geometriques 
associees  sont  des  segments,  des  ensembles  de  points  et 
des  chaines. 

Le  traitement  consiste  a  decrire  des  elements  du 
contexte  selectionnes  et  la  cible  a  l’aide  de  ces  fonnes 
geometriques.  Cette  description  constitue  le  modele  de 
reference  qui  est  etabli  en  preparation  de  mission  a 
l'aide  de  donnees  cartographiques  ou  d'images.  Pendant 
la  mission  le  senseur  image  le  sol  et  le  traitement  extrait 
de  1’ image  ces  elements  puis  les  met  en  correspondance 
avec  ceux  du  modele. 

Le  resultat  du  traitement  est  triple: 

-  recalage  de  l’image  senseur  (et  done  de  l’avion)  par 
rapport  au  terrain, 

-  aide  a  l’identification  du  contexte  par  plaquage  du 
modele  de  reference  sur  l’image  senseur  recalee, 

-  reconnaissance  de  cibles  etendues. 

Si  les  coordonnees  relatives  d’un  objectif  sont  connues 
par  rapport  au  terrain  decrit  par  le  modele  de  reference, 
le  recalage  image  pennet  d’avoir  automatiquement  sa 
localisation  par  rapport  a  l’avion. 

Ce  principe  de  mise  en  correspondance  devrait  etre 
utilise  pour  la  mise  en  correspondance  des  images  radar 
et  optronique. 

Les  traitements  permettant  de  detecter  et  de  recomiaitre 
directement  des  cibles  de  petites  tail  les  sont  a  etudier. 

II  existe  deux  grandes  families  pour  la  detection: 

-  les  traitements  qui  reconnaissent  le  fond  et  qui  par 
soustraction  detectent  les  cibles, 

-  les  traitements  qui  utilisent  une  description  des 
cibles  (formes,signatures)  pour  les  detecter. 

Les  traitements  du  second  type  sont  egalement  utilises 
pour  la  reconnaissance. 


THOMSON-CSF  a  deja  etudie  ou  realise  des 
applications  (de  classification  de  radars  ou  de  cibles  air 
menaijantcs  par  exemple)  utilisant  des  techniques 
neuronales,  de  segmentation  statistique  ou  d'agregation 
multi-criteres. 

Parmi  les  difficultes  de  l’application  de  detection  et  de 
reconnaissance  de  cibles  sol  imagees  on  peut  citer: 

-  l’apprentissage  qui  devra  englober  un  tres  grand 
nombre  de  cas  (cibles,  fonds,  conditions  climatiques 
et  meteorologiques,...), 

-  l’agregation  des  informations  partielles  resultant  des 
differents  modes. 

Les  travaux  a  venir  devront  egalement  considerer  de 
quelle  fai^on  les  infonnations  de  DRI  issue  d’un  mode 
pourront  etre  utilisees  avec  un  autre  et  ceci  dans  deux 
hypotheses  d’architecture: 

-  aval  ou  le  systeme  agrege  des  infonnations  de  DRI 
issues  de  traitements  locaux  effectues  par  chaque 
senseur, 

-  amont  ou  le  systeme  traite  directement  les  images. 

5-  Les  applications 

Les  principals  applications  considerees  dans  le  cadre 
de  ces  travaux  sont: 

-  le  recalage  de  navigation, 

-  la  designation  d’un  objectif  sol:  detecter,  recomiaitre 
puis  localiser  l’objectif  (si  necessaire), 

-  le  traitement  d’une  cible  par  un  AASM  avec  un 
senseur  a  imagerie  initialise  a  partir  d’images 
acquises  par  les  senseurs  de  bord. 

Nous  analyserons,  dans  le  cadre  de  ces  travaux,  les 
possibility  du  systeme  en  fonction  des  exigences  de 
l'AASM  rappel ees  dans  le  §1  (portees,  precisions, 
emploi  jour/nuit,  toutes  altitudes,  capacite 
multicible,...). 

6-  Conclusions 

Des  travaux  sont  en  cours  pour  disposer  d’un  armement 
tactique  precis  (metrique  ou  decametrique)  aux  portees 
accrues  afm  d’ameliorer  la  survivabilite  et  l’efficacite 
de  l’avion  dans  un  enviromiement  complexe.  La  mise  en 
oeuvre  de  cet  armement  necessite  de  disposer  en  amont 
de  capteurs  permettant  de  detecter,  de  reconnaitre  et 
de  localiser  precisement  les  cibles.  La  disposition  de 
ces  capteurs  a  bord  de  l’avion  ameliore  grandement 
les  capacites  du  systeme  en  matiere  de  reactivite 
(allegement  de  ia  preparation  de  mission)  et  de 
traitements  des  cibles  mobiles  ou  depla^ables. 

C’est  pourquoi,  en  parallele  de  travaux  consacres  a 
1  armement,  la  DGA  a  lance  des  travaux  relatifs  aux 
modes  Air-Sol  des  senseurs  du  RAFALE. 

Ces  travaux  ont  deux  finalites: 

1-  Avoir  de  modes  d’imagerie  radar  et  optronique  tres 
perfonnants. 

2-  Etre  en  mesure  d'exploiter  leur  complementarite  par 
des  principes  d’emploi  et  une  assistance  a 
l’exploitation  pour  detecter,  recomiaitre  et  localiser 
(systeme  complexe  et  enviromiement  dense  et 
evolutif) 
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II  est  tres  important  de  maitriser  la  chaine  qui  va  de  la 
collecte  de  P  information  (la  reconnaissance)  an 
traitement  de  la  cible  (par  Panne)  en  passant  par  la 
confirmation  et  Paffinage  des  parametres  de  designation 
d'objectif  (par  les  capteurs  des  avions  dn  raid). 


L’interet  de  cette  maitrise  est  double  : 

-  garantir  Phomogeneite  du  systeme  d’armes  et 
proposer  la  meilleure  solution  a  cout  minimal, 

-  rationaliser  le  traitement  d'infonnation  capteurs  pour 
les  fonctions  de  detection  et  de  reconnaissance  qui 
existent  au  niveau  reco,  avion  et  anne. 
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SUMMARY 

A  system  is  described  by  which  on  the  basis  of  IR-signatures 
ships  are  classified  using  methods  of  varying  interactivity.  A 
sequence  of  images  taken  with  an  IR-camera  in  a  sea 
environment  is  superimposed  so  that  pixels  in  corresponding 
frames  match  as  closely  as  possible.  With  this  motion 
compensation  and  a  following  long-term  filtering  the  signal-to- 
noise  ratio  is  significantly  improved. 

Subsequently  the  contour  lines  of  the  ship  are  extracted  by  a 
gradient-based  or  an  area-based  algorithm.  For  sequences  with 
a  good  signal-to-noise  ratio  this  can  be  done  automatically  in 
other  cases  an  interactive  component  is  necessary. 

The  contour  of  the  "unknown"  ship  is  the  feature  foi  a 
correlative  or  a  neuronal  classifier.  For  both  classifiers  the 
reliability  and  the  computing  time  is  strongly  dependent  upon 
the  size  of  the  reference  database.  It  is  of  inestimable  benefit  to 
include  all  the  available  sensor  data  (e.g.  sonar  and  radar)  in  the 
classification  process  thus  quickly  and  significantly  reducing 
the  initially  very  large  search  space. 

LIST  OF  SYMBOLS 

7i  image  plane 

7t0  object  plane 

xJcYit  point  coordinates  in  image  plane 

x0,y0  point  coordinates  in  object  plane 
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d  distance  between  origin  on  image  and  object  plane 
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s^  length  between  characteristic  points  in  7t 
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s  normalized  length  s 

S  sum  of  normalized  lengths  sj 

tor  weight  of  neuron  at  location  r 

CO/  weight  of  winner  neuron  at  location  r' 

m  input  vector 

e  learning  coefficient 

hrr-  neighbourhood  function 

q  width  of  neighbourhood  function 

P  ’wavering’  product 

skn  nlh  silhouette  of  the  k,h  reference  image 

hk(x,y)  frequency  of  the  kth  reference  image  at  the  spot  (x,y) 

N  number  of  silhouettes  in  the  kth  group 

hk  reference  image 
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s i  database  silhouette 

s2  silhouette  of  the  object  to  be  classified 

1  INTRODUCTION 

Even  for  a  skillful  operator  it  is  a  tedious  and  lifetime  labor  to 
classify  unknown  ships  with  the  aid  of  a  large  database  in  the 
form  of  a  book  e.g.  with  Jane's  catalog  of  Fighting  Ships. 

Using  the  latest  CD-ROM  version  of  this  catalog  accelerates  the 
classification  process,  but  the  interactive  component  of  the 
operator  is  furthermore  very  high. 

Therefore  the  question  arises:  Can  the  classification  be  done 
partly  or  fully  self-acting?  In  this  paper  a  method  is  proposed  to 
classify  "unknown"  ships  in  a  maritime  scene.  Depending  on 
the  signal-to-noise  ratio  of  the  IR-image  the  classification  can 
be  done  automatically  or  an  interactive  component  of  an 
operator  is  necessary. 

At  first  in  chapter  2,  a  computer-aided  interactive  classifier  will 
be  described.  Then  in  the  subsequent  chapter  3  motion- 
compensation  and  long-term  filtering  is  discussed.  This  is  the 
basis  for  the  feature  extraction,  which  is  discussed  in 
connection  with  the  correlative  classifier  in  chapter  4  and  with 
the  neuronal  classifier  in  chapter  5.  In  both  chapters  it  is  easily 
seen  that  all  available  sensor  data  will  accelerate  and  ameliorate 
the  classification  process.  With  a  short  conclusion  in  chapter  6 
this  paper  will  be  terminated. 

2  A  COMPI JTER  AIDED  INTERACTIVE  CLASSIFIER 


The  problem  of  recognition  of  an  unknown  ship  can  be  led  back 
to  the  basic  scheme  of  pattern  recognition  shown  in  Figure  2-1. 


Unknown 

Object 

> 

Sensor 
IR/  Video 

feature 

Extraction 

Classifier 

Database 

Figure  2-1:Basic  scheme  of  a  pattern  recognition  system 

The  object  to  be  classified  (OTBC)  is  filmed  by  a  sensor  (IR  or 
Video).  After  extracting  characteristic  features  these  are 
compared  with  the  corresponding  items  in  the  database  and  the 
unknown  object  is  classified. 

In  many  cases,  due  to  the  low  signal-to-noise  ratio  the 
automatic  feature  extraction  is  impossible.  Therefore,  under 
these  bad  conditions,  a  human  operator  carries  out  the  feature 
extraction. 

To  what  extent  a  special  feature  is  significant  for  the 
classification  process  cannot  be  answered  easily.  In  this  chapter 
the  feature  "ship  parts"  and  their  position  to  one  another  have 
proven  significant.  Figure  2-2  shows  an  infrared  picture  of  a 
ship.  The  crosses  mark  the  x  and  y  coordinates  of  the  bow,  the 
masts,  the  bridge,  the  funnel  and  the  stem  in  the  image. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar",  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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The  Euclidean  distances  between  the  different  individual  "ship 
parts"  and  the  bow  are  scaled  to  the  distances  between  bow  and 
stem.  These  numbers  are  the  features  of  the  OTBC. 

A  database  contains  the  same  data  for  all  models  to  be 
classified.  Therefore,  a  classifier  compares  the  data  of  the 
OTBC  with  the  corresponding  data  of  each  individual  model. 
The  difference  between  two  corresponding  "ship  parts"  is  a 
measure  of  the  similarity  (distance  near  zero)  or  dissimilarity 


Figure  2-2:  Infrared  picture  of  a  ship 


(distance  very  great)  of  the  OTBC  and  the  chosen  model  of  the 
database. 

Having  compared  the  OTBC  with  all  models  one  has  a  number 
for  each  comparison  (OTBC-model).  Sorting  these  numbers  in 
a  rising  series  and  using  the  similarity  as  a  feature,  the  model 
with  the  lowest  number  has  the  highest  probability  to  be  the 
OTBC. 

Furthermore  the  operator’s  statement  that  he  has  chosen  a 
particular  "ship  part"  is  included  in  the  classification  process  in 
a  fault-tolerant  way. 

Confusion  of  a  mast  with  a  gun  is  much  less  probable  than  that 
of  a  gun  with  a  rocket  launcher.  Therefore  each  pair  of  "ship 
parts"  is  allocated  a  confusion  probability  and  derived  from  this 
probability  a  number  that  is  added  to  the  already  calculated 
Euclidean  distances.  Improbable  assignments  result  in  great 
distances. 

Thus  the  total  distances  imply  the  aberration  of  the  geometrical 
distances  and  the  confusion  in  the  determination  of 
corresponding  "ship  parts"  in  a  fault  tolerant  way. 

In  accordance  with  the  number  of  "ship  parts"  and  models  in 
the  database  the  time  necessary  for  the  calculation  of  all 
possible  distances  between  the  OTBC  and  the  models  increases 
exponentially.  (NP  problem)  With  the  so-called  A*  algorithm  a 
perhaps  suboptimal  solution  is  found  within  acceptable  time. 
Furthermore  leaving  extraordinary  meteorological  situations  out 
of  account  (pitch  and  roll-angles  are  negligible)  the  position 
angle  is  directly  connected  to  the  visibility  of  the  individual 
"ship  parts".  Thus  the  information  of  visibility  must  also  be 
stored  in  the  database. 

Preliminary  tests  with  a  database  of  200  models  and  7  to  8 
marked  "ship  parts"  showed,  that  the  model  corresponding  to 
the  OTBC  was  found  on  the  average  among  the  first  5  ships  of 
the  list  arranged  to  decreasing  similarities. 

A  visual  comparison  of  the  OTBC  and  the  model  images 
verifies  and  completes  the  recognition  process. 

Further  details  are  described  in  (Ref.  1) 

3  MOTION-COMPENSATION  AND  EONG-TERM 
FILTERING 


The  image  quality  of  thermal  imagers  available  today  is  bad 
with  regard  to  those  of  the  visual  spectrum.  The  IR-images 
suffer  from  a  low  signal-to-noise  ratio,  low  contrast  at  the 
object  boundaries,  the  propagation  conditions  and  sensor 
artifacts.  Therefore  the  usual  steps  in  image  processing  are  not 
sufficient  enough  to  guarantee  almost  error  free  results. 

A  good  basis  for  preprocessing  and  extraction  of  the  contour 


line  as  a  feature  relevant  for  classification  is  the  so-called 
motion-compensated  sequence  of  IR-images.  These  sequences 
have  much  more  data  at  their  disposal  than  a  single  image  and 
therefore  the  evaluation  security  increases  in  a  significant 
statistical  sense. 

With  the  aid  of  "time  information"  new  evaluation  algorithms 
adapted  to  these  problems  can  be  formulated. 

As  shown  in  Figure  2-1  and  Figure  3-1  a  sequence  of  the  OTBC 
is  taken  from  a  sensor  platform.  OTBC  and  sensor  platform  are 
both  in  relative  motion  to  one  another.  Thus  it  is  not  possible  to 
take  directly  the  mean  of  all  images  in  the  sequence,  to 
ameliorate  the  signal-to-noise  ratio. 

At  first  both  motions  must  be  estimated  and  then  compensated 
to  get  a  stationary  object  that  does  not  move  relative  to  the 
image  border.  Then  the  mean  value  of  the  images  of  the 
sequences  is  formed. 


Figure  3-1:  Motion  compensation  of  image  sequences 
The  main  items  of  this  procedure  are  explained  in  Figure  3-1. 

In  a  first  step  a  noise  measurement  is  made  in  an  image 
sequence.  This  is  the  basis  for  the  adaptation  of  several  modules 
to  image  quality,  which  changes  significantly  due  to 
meteorological  situations,  object  attributes  and  distance 
between  sensor  and  OTBC. 

Then  the  motion  compensation  of  the  OTBC  and  the  sensor 
platform,  which  are  subjected  to  three-dimensional  motion,  are 
carried  out. 

It  is  necessary  to  know  scene  area  indicators,  which  can  be 
clearly  assigned  to  the  object  (OTBC)  and  to  the  sensor 
platform. 

Hot-spots  are  well  suited  for  this  purpose  and  they  belong  to  the 
OTBC.  Therefore  a  hot-spot  detector  module  was  realized, 
which  determines  all  hot-spots  in  the  images  of  a  sequence.  In  a 
spatio-temporal  filter  the  hot-spots,  e.g.  sunglints,  which  do  not 
belong  to  the  OTBC,  are  eliminated. 

A  scene  area  indicator  for  the  sensor-platform  ego-motion  is  the 
horizon,  which  can  be  determined  in  nearly  all  images  without 
great  difficulty. 

Both  methods  serve  as  the  basis  for  an  Extended  Kalman  filter, 
by  which  a  motion-compensated  calculation  of  a  mean  value 
image  is  realized,  that  means  that  the  OTBC  in  all  images  of  the 
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scene  is  shifted  so  that  its  position  relative  to  the  image  border 
is  fixed.  (Ref.  2, 3, 4) 

Thus  the  signal-to-noise  ratio  of  the  mean  value  image  is 
significantly  improved  depending  on  the  square  root  of  the 
length  of  the  sequence. 

The  result  of  motion-compensation  is  illustrated  in  the  Figure  3- 
2,  where  a  single  image  of  the  sequence  and  the  mean  value 
image  are  shown.  To  demonstrate  the  amelioration  of  the  mean 
image  details  of  the  ships  are  magnified. 

The  basis  for  the  following  contour  extraction  are  the  mean 
value  image  and  all  the  images  of  a  scene  which  will  be  shown 
in  the  next  chapter. 


Figure  3-2:  top:  Single  image  of  a  sequence 
bottom:  Mean  value  image 


4  FEATURE  EXTRACTION  AND  THE  CORRELATIVE 
CLASSIFIER 


In  the  interactive  classification  procedure  in  chapter  2  the  "ship 
parts"  and  their  positions  are  used  as  features  for  the  classifier. 
For  the  automatic  and  semiautomatic  classification  process  the 
contour  of  the  OTBC  proved  to  be  a  powerful  feature. 

Therefore  besides  the  interactive  approaches  two  automatic 
techniques  of  object  contour  extraction  have  been  studied. 

These  differ  from  the  wellknown  procedures  in  literature,  as 
they  take  into  account  the  time  information  from  motion 
compensated  image  sequences. 

The  first  method  uses  primarily  distinguishing  criteria  for 
different  temporal  and  spatial  local  gray-level  histograms, 
whereas  the  second  method  is  especially  based  on  homogeneity 
criteria. 

Both  procedures  can  be  regarded  as  extensions  of  the  contour- 
based  or  region-oriented  segmentation  algorithms  known  in 
literature  for  at  least  a  couple  of  years. 

The  first  method  of  contour  extraction  is  divided  into  two 
moduls.  In  one  modul  stationary  contour  areas  are  extracted  and 
in  a  second  module  an  adaptive  hierarchical  contour  extraction 
with  a  subsequent  rule-based  algorithm  is  performed. 

With  the  aid  of  a  comparison  of  the  local  and  temporal  variance 
of  the  compensated  sequence,  stationary  structures 
corresponding  to  the  OTBC  and  instationary  structures 
corresponding  to  the  water-surface  texture  are  separated. 
Stationary  object  contours  appear  only  in  regions  where  the 


local  variance  is  significantly  higher  than  the  time  variance. 
Therefore  the  quotient  from  the  spatial  variance  of  a  local 
window  of  the  time  averaged  image  sequence  and  from  the  time 
variance  of  this  image  sequence  locally  averaged  in  this 
window  is  evaluated.  Three  classes  can  be  roughly  assigned  to 
this  quotient: 

1 .  In  the  sky  and  in  the  internal  object  areas  the  gray  levels  are 
due  to  Gaussian  noise  and  using  a  suitable  scaling  the  variance 
quotient  obeys  a  F-distribution. 

2.  The  variance  quotient  in  the  area  of  the  water  surface  also 
responds  to  a  F-distribution,  but  the  expected  value  is  shifted 
towards  smaller  values. 

3.  In  the  neighbourhood  of  the  contour  the  variance  quotient  is 
larger  and  its  distribution  is  unknown. 

With  these  conditions  in  mind  appropriate  threshold  values  for 
stationary  contours  can  be  found.  The  security  of  this  procedure 
is  improved  by  a  hierarchical  application  to  a  spatial/time 
subsampled  motion-compensated  image  sequence. 

Subsequently  an  adaptive  hierarchical  contour  extraction  is 
carried  out  for  the  stationary  contour  areas.  This  helps  to 
determine  the  fine-structure  of  the  OTBC-contour.  The  result  of 
the  contour  based  segmentation  in  a  motion-compensated  and 
long-term  filtered  image  sequence  is  shown  in  Figure  4-1. The 


Figure  4-1 :  Result  of  the  contour-based  segmentation, 
good  performance  of  this  technique  is  noteworthy. 

The  second  method  of  contour  extraction  is  a  region-oriented 
segmentation  approach,  which  is  based  on  an  extended  split  & 
merge  algorithm.  An  extended  set  of  characteristics  is  used  in 
the  split  -  as  well  as  in  the  merge  phase.  It  is  also  based  on  the 
inclusion  of  different  spatial  and  temporal  statistics  of  the 
motion-compensated  image  sequence. 

To  determine  image-adapted  decision  thresholds  an  artificial 
neural  network  is  implemented  in  this  method.  Furthermore  the 
knowledge  of  the  source  is  employed  in  the  segmentation 
procedure. 

In  the  split  process  the  image  is  stepwise  divided  in  details 
which  can  be  regarded  as  homogeneous  with  regard  to  a  special 
feature. 

In  the  merge  phase  homogeneous  neighbouring  base  segments 
are  grouped  together,  if  they  are  sufficiently  similar. 

Regionally  global  as  well  as  regionally  local  criteria  are  used 
for  similarity  or  dissimilarity  of  regions. The  base  segments  and 
the  regions  are  iteratively  connected  whereby  the  decision 
threshold  is  lowered  step  by  step. 

In  a  last  step,  called  interpretation  phase,  the  combination  of 
regions  is  completed  with  regard  to  plausibility  criteria.  Then 
on  the  bases  of  rules  and  of  a  scene  model  the  regions  are 
assigned  to  background  and  object. 

Finally  the  outer  border  of  the  OTBC  is  determined  which 
corresponds  to  its  contour. 

The  result  of  the  region-based-algorithm  in  a  motion- 
compensated  and  long-term  filtered  image  sequence  is  shown  in 
Figure  4-2. 


Figure  4-2:  Result  of  the  region-based  segmentation 


With  this  just  mentioned  method  one  receives  always  a  closed 
contour,  which  is  the  direct  or  indirect  basis  for  the  different 
classifiers.  A  combination  of  both  segmentation  methods  is 
under  development. 

4. 1  Correlative  Procedure 

By  means  of  the  correlative  classifier  the  object  to  be  classified 
is  assigned  to  available  3D  ship  models.  In  this  approach  the 
correlation  coefficient  serves  as  a  resemblance  criterion.  By 
way  of  example,  the  structure  of  a  classifier  is  presented  with 
the  aid  of  two  correlation  criteria  (Equation  4-2  and  Equation  4- 
3). 

Figure  4-3  shows  the  structure  of  the  correlative  classifier. 
Classification  is  done  in  two  steps  designated  as  classifiers  I 
and  II  in  Figure  4-3. 

4.2  Comparative  Data 

For  the  purpose  of  classification  the  object  to  be  classified  is 
compared  with  models  from  a  database.  The  database  is 


Figure  4-4:  top:  Silhouette 

centre:  Roughly  quantized  binary  matrix 
bottom:  Gray-level  reference  image 

From  the  binary  silhouettes  produced  that  way,  a  cluster 
algorithm  determines  a  restricted  number  of  reference  images 
each  of  which  represents  a  group  of  ship  silhouettes  similar  in 
appearance  (Figure  4-4  bottom).  The  relative  frequencies  of  a 
pixel  occurring  at  the  spot  (x,  y)  of  all  N  silhouettes  belonging 
to  one  group  are  stored  in  the  matrix  elements  of  these 
reference  images  according  to  Equation  4-1. 

N 

Y,Skn(x’y)  Equation  4-1 

M*,y)  =  ^_ - 

The  binary  matrix  and  the  reference  image  matrix  are  the 


Figure  4-3:  Structure  of  the  classifier 


established  according  to  the  following  scheme.  On  the  basis  of 
3D-ship  models,  2D-silhouettes  (Figure  4-4  top)  are  produced. 

The  3D-models  are  looked  at  under  a  specific  role-,  pitch-  and 
position  angle.  Only  the  silhouettes  that  may  appear  in 
dependence  upon  the  sensor  location  are  stored  in  the  database 
and  supplied  to  a  cluster  algorithm. 

The  database  silhouettes  are  roughly  quantized.  The  gray  levels 
are  binarized  and  the  length  of  the  OTBC  is  normalized  to  32, 
48,  or  64  points  (pixels).  The  result  is  a  binary  matrix  shown  in 
Figure  4-4,  centre. 


comparative  data  used  for  classification. 

4.3  Classification 

The  ship  silhouette  provided  by  means  of  feature  extraction  is 
also  roughly  quantized  for  classification.  The  silhouette  of  the 
object  to  be  classified  is  now  available  as  a  binary  matrix. 
Regarding  classification  it  is  of  no  importance  whether  an 
automatically  produced  or  a  manually  produced  silhouette  is 
used. 

In  both  cases  the  OTBC  is  available  as  a  binary  matrix.This 
matrix  is  classified  by  means  of  classifier  I  depicted  in  Figure 
4-3.  In  a  first  step,  the  binary  matrix  of  the  OTBC  is  assigned  to 
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one  or  more  reference  images  stored  in  the  database,  that  is,  to 
one  or  more  classes  of  silhouettes  according  to  the  maximum  of 
Equation  4-2 


n- 1  m-l 

X  X  hk  (*’  'v)  •S2<-X  +  Vx-  y  +  vy  > 

n\s2  =  F0p0  7m -  Eciuation  4'2 

XXi2(x'v) 

x=0  y=Q 

0 < hk (x, y) <  1 ,  s2(x,y)e (0,1) 

After  this  first  step,  statements  can  be  made  and  issued  on  the 
probable  group  and  the  position  angle  with  regard  to  the 
observer.  A  fault  tolerant  filter  selects  the  binary  silhouettes 
corresponding  to  the  group  and  to  the  position  angle  from  the 
2D-database.  They  are  used  for  classification  in  classifier  II. 

In  the  second  classifier  it  is  attempted  to  assign  the  silhouette  of 
the  object  to  be  classified  unambiguously  to  a  database 
silhouette.  For  this  purpose  an  average-free  normalized  cross¬ 
correlation  function  is  used  (Equation  4-3).  In  some  cases  it  is 
insufficient  to  calculate  the  correlation  coefficient  only  with  the 
exact  pixels  of  the  OTBC  and  the  database  silhoutte.  An 
operator  wich  includes  neighbouring  pixels  provides  better 
results.  For  this  purpose  the  second  classifier  uses  the  operator 
Okkfmfb  t0  broaden  the  silhouettes. 


OfCKFMFB  - 
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In  both  classifications  the  silhouette  of  the  object  to  be 
classified  (S2)  is  moved  by  vx  and  vy  on  the  corresponding 
comparative  image  until  the  maximum  correlation  coefficient  is 
found. 


decreasing  matrix  width.  Too  great  a  number  of  matrix  columns 
is  also  pointless  as  very  high  accuracy  requirements  are  to  be 
established  regarding  the  3D-models  and  the  extracted 
silhouettes.  Furthermore,  the  amount  of  computation  increases 
quadratically  with  increasing  the  matrix  width  and  hight  by  the 
same  factor. 


Figure  4-5:  Results 

top:  Automatically  extracted  silhouette 
bottom:  Manually  extracted  silhouette 

In  conclusion  it  can  be  stated  that  classifier  I  ensures  a 
silhouette  assignment  of  almost  100%  to  the  right  silhouette 
groups  and  classifier  II  achieves  an  identification  rate  of  80%  of 
the  ship  silhouettes.  However,  it  is  to  be  noted  that  these  results 
refer  to  manually  established  silhouettes  and  to  currently  used 
databases  comprising  approximately  100  3D-models.  The 
classification  and  identification  results  obtained  with 
automatically  extracted  ship  silhouettes  are  strongly  dependent 
on  the  quality  of  the  silhouettes  provided. 

5  NEURAL  SHIP  RECOGNITION  FROM  PERSPECTIVE 
IMAGES  BASED  ON  THE  CONTOUR 


The  ‘correlation  coefficients’  of  Equation  4-2  and  Equation  4-3 
are  between  0  and  1 . 

In  the  final  step  the  correlation  coefficents  are  sorted  in  a  rising 
series.  For  unambiguous  classification  it  is  necessary  to  carry 


In  chapter  3  the  motion  compensation  of  an  infrared  image 
sequence  was  explained.  Motion  compensation  is  necessary  in 
order  to  extract  the  ship  from  the  image  background  as  shown 
in  chapter  4.  Then,  finally,  the  contour  of  the  ship  is  obtained, 
which  is  the  feature  to  identify  the  ship. 


/i-l  m-l 

XXlM^-lM  -{^U  +  v^y  +  vp-^O^y) 


KKFMFV2(vx,vy)  = 


x=0  y=0 


n- 1  m-l  n- 1  m-l  ^ 

S  (x,y)-r1(x,y  Till-  2(x,y)-j2(x,y)} 
y x=0  v=0  x=0  v=0 


Equation  4-3 


Ji(x,y)e(0,l),  s2(x,y)e(0,l) 

out  a  visual  comparison  of  the  object  to  be  classified  (OTBC) 
with  the  database  silhouettes.  Here,  the  operator  is  supported  by 
the  computer  laying  the  database  silhouette  assigned  to  the 
OTBC  and  the  OTBC  silhouette  on  top  of  one  another  (Figure 
4-5).  The  database  silhouettes  are  represented  in  Figure  4-5  by 
gray  squares. 

4.4  Results 

Extensive  tests  have  been  carried  out  with  the  classifiers 
described.  Good  results  are  obtained  if  the  spatial  resolution  of 
the  matrix  (Figure  4-4)  is  well  adapted  to  the  dimension  of  the 
OTBC  in  the  image.  The  loss  of  information  increases  with  a 


This  identification  is  performed  by 

-  extracting  characteristic  points  of  the  contour,  which  are 
found  independent  of  the  viewing  position, 

-  calculating  scale  invariant  features  between  the  characteristic 
points  for  contours  of  many  different  viewing  positions, 

-  training  a  self-organizing  feature  map  with  these  features, 
which  does  the  identification  after  the  learning  process. 

This  identification  method  then  provides  the  feasibility  of  ship 

recognition  on  the  basis  of  one  contour  only. 
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5. 1  Extraction  of  Characteristic  Points 


The  contour  of  a  ship  in  the  image  plane  7t  ( In  this  chapter  in 
the  contrary  to  the  previos  one  two  different  coordinate  systems 
with  the  indices  Tt  and  0  are  used)  is  an  aggregate  of  points, 


Figure  5-2:  Characteristic  points  (*)  of  the  contour  under 
0  =  5°  and  inverse  projected  characteristic  points  (o)  of 
contour  in  Figure  5-3 


Figure  5-3:  Characteristic  points  of  contour  under  0  =  65° 
5.2  Scale  Invariant  Feature  Calculation 


Figure  5-1:  Relation  between  object  and  image  plane 

which  are  projected  with  slant  0  (see  Figure  5-1)  from  the  ship 
in  3D  space  into  the  2D  image  plane.  Hence,  the  shape  of  the 
contour  is  dependent  on  the  camera’s  viewpoint.  Fortunately, 
this  aggregate  of  points  contains  some  points,  which  can  be 
found  independent  of  the  viewing  position  at  constant  values  of 
normalized  yK  -coordinate,  if  some  assumptions  are  fulfilled. 
These  assumptions  are 

-  long  distances  d  between  the  ship  and  the  camera 

-  and  the  focal  length  f  »  x.K,yK. 

Using  these  assumptions  the  formulas  of  inverse  projection 
from  image  plane  n  into  object  plane  7t0,  as  they  read. 


d{x^  cos(x  )  +  yK  sin(x)) 
cos  (0)Dj- 

d(-xKsin(x)  +  ylzcos(x)) 

Dj' 


Equation  5-1 


Equation  5-2 


with 

DT  =  (f  +  xn  tanfGjcosCO  +  yjj  tan(0)sin(x)) , 


Now,  the  length  s0  (0)  between  the  characteristic  points  are 
modeled  for  all  possible  viewpoints  between  0°<  0  <  65°. 


sQm  =  +s%s/cos2Q. 


Equation  5-5 


If  this  length  in  the  image  as  well  as  in  the  object  plane  is 
normalized  by  proper  perimeter  of  the  contour,  then  it  can  be 
approximated  using  Taylor  expansion  (Ref.  6): 


iO(0)  =  ^  +a?-02 


Equation  5-6 


This  approximation  also  holds  for  an  integral 
i 

length,  S,  =  ,  and  can  as  well  be  deduced  for  the  angles 

7=0 

between  the  contour  segments. 

These  relations  between  integral  length  (angles)  and  slant  0 
are  used  to  generate  the  training  data  for  the  neural  network  on 
the  basis  of  one  contour  per  ship.  The  neural  network  is  a  self¬ 
organizing  feature  map,  which  has  to  identify  later  on  contours 
obtained  from  3D  models  of  ships  and  noisy  contours  extracted 
from  infrared  image  sequences. 


are  simplified  to 

x0  =  (Xn  cos(T)+yJC  sin(x))/cos(0)  Equation  5-3 

y0=-x7:sin(T)  +  yIccos(T).  Equation  5-4 

Furthermore,  in  standard  recognition  situations  the  sea  is 
calm,  so  that  tilt  x  is  small  (see  Equation  5-3  and 
Equation  5-4).  Then,  normalizing  the  y„  -coordinate  to 
the  height  of  the  ship,  characteristic  3D  points  are 
projected  to  image  points  lying  on  a  special  hull  of  the 
contour  (see  Figure  5-2)  at  constant  normalized  values  of 

y*- 


5.3  Topographic  Presentation  with  SOM 

The  neural  network  algorithm  of  Kohonen  (Ref.7,  8)  produces 
some  topographic  presentation  of  the  training  data  in  a  space  of 
lower  dimension  than  the  input  training  data  space.  A 
topographic  presentation  of  the  training  data  is  desired  in  order 
to  classify  even  unknown  ships  or  hardly  identifiable  ones. 

Kohonen’s  algorithm  can  be  summarized  describing  the 
learning  rule  for  the  weight  vector  tor  at  location  r  by 
Equation  5-7  and  Equation  5-8. 

Ksrij)  =  ro  r(j- 1 )  +  Aw  tij)  Equation  5-7 

Am  tij)  =  e(/ )  hrr’U  )[-»i(j)-m,{j-l) 


Equation  5-8 
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Equation  5-8  contains  hrr'(j ),  which  is  a  Gaussian 
neighbourhood  function.  Its  maximum  is  at  the  location  /  on 
the  feature  map. 


( 

hrr'ij  )  =  exp  • 

V 


\r-r> 


t  |2 


2<;20 ')j 


Equation  5-9 


with 

I  r~i’\  =A/(ri-ri')2+-+(rn-rM')2 

This  location  is  found  by  calculating  the  nearest  neighbour  to 
the  input  vector  m: 

lay  -m  I  =  min  l©r  -  m  I  Equation  5-1 0 


Figure  5-4:  Ideal  organization  of  ships  on  a 
feature  map  (0°  <  0<  ±  90°) 


feature  map  ideally  the  training  data  should  be  organized  in  a 
way  shown  in  Figure  5-4.  In  fact,  a  map  of  this  kind  (see  Figure 
5-5)  is  obtained  including  4  ships  in  the  data  set.  Selecting  18 
ships  topology,  is  best  preserved  on  four  dimensional  maps. 
This  is  concluded  from  the  wavering-product,  which  is  a 
measure  of  topology  preservation  on  a  feature  map  (Ref.  6) 


Neurons  in  each 
dimention 

Wavering- 

product 

3D-model 

contours 

(O°<0<65°) 

Infrared- 

contours 

27x27 

-0.0221 

84% 

55% 

9x9x9 

-0.0043 

80% 

45% 

5  x  5  x  5  x  5 

-0.0020 

81% 

30% 

9  x  9  x  9  x  9 

2.5- 10-4 

81% 

35% 

5x5x5x5x5 

0.0255 

81% 

50% 

Table  5-1:  Identification  results  on  various  feature 
maps  containing  18  ships 


The  wavering-products  of  trained  higher  dimensional  feature 
maps  are  shown  in  Table  5-1.  A  value  near  zero  indicates  good 
topology  preservation  (Ref.  6).  This  is  achieved  in  case  of  four 
dimensional  feature  maps.  In  case  of  four  dimensional  maps  the 
identification  rate  of  contours  from  3D-models  is  81%  .  Only 
35%  of  the  infrared  contours  are  identified.  Surprisingly,  lower 
dimensional  maps  show  better  identification  results.  84%  of 
the  3D-modcl  contours  and  55%  of  infrared  contours  are 
identified.  It  is  concluded,  that  after  some  parameter  studies 
four  dimensional  maps  can  be  trained  to  reach  as  good 
identification  rates  as  in  case  of  the  two-dimensional  map. 

In  summary,  the  presented  contour  based  method  to  recognize 
ships  from  various  viewpoints  shows  good  results  on  the  chosen 
data  set  of  18  ships.  Its  robustness  is  demonstrated  by  the 
infrared  contour  identification  rate  of  55%  in  a  two- 
dimensional  feature  map.  In  contrast,  an  early  proposed  method 
of  viewpoint  invariant  ship  recognition  using  higher  order 
neural  networks  (Ref.  9)  is  not  as  successful  as  this  method, 
though  it  uses  9  different  contours  of  the  ships  during  the 
learning  process. 


Analyzing  this  algorithm’ s  behaviour  as  well  as  that  of  the  The  proposed  neural  identification  method  is  able  to  identify 

training  data,  it  can  be  deduced,  that  on  a  two-dimensional  ships  on  the  bases  of  their  contours.  Because  the  shape  of  the 


v- coordinate 


u-coordinate 
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contour  is  viewpoint  dependent,  it  is  worth  mentioning,  that  the 
proposed  method  is  feasible  to  identify  a  ship  if  only  one  of  its 
contours  is  known.  In  contrast,  other  methods  need  large 
databases  of  possibly  occurring  contours,  which  have  to  be 
laboriously  collected.  This  advantage  is  achieved  by  extracting 
characteristic  points  on  a  special  defined  hull  of  the  contour 
invariant  of  the  viewpoint.  Afterwards,  length  and  angles 
between  these  points  are  modeled  for  all  viewpoints  and  then 
used  to  train  a  self-organizing  feature  map.  Its  identification 
rates  reach  84%  in  case  of  good  contours,  55%  in  case  of 
blurred  ones. 

These  results  are  very  good  in  comparison  to  other  approaches 
of  viewpoint  invariant  ship  identification  (Ref.  9). 

6  CONCLUSION 
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System  for  Infrared  Image  Sequence,  Analysis  and 
Classification.  Proceedings  IASTED  International 
Conference  on  Signal  and  Image  Processing,  pp.  229-232, 
1995. 

10.  Pizlo,  Z.;  Rosenfeld,  A.,  Recognition  of  Planar  Shapes  from 
Perspective  Images  Using  Contour-Based  Invariants.  Image 
Understanding,  56(3):330— 350,  November  1992. 


In  this  paper  different  methods  of  classifying  IR-signatures  of 
ships  have  been  discussed.  Depending  on  the  signal-to-noise 
ratio  of  the  IR-images  the  classification  can  be  done 
automatically  or  an  interactive  component  is  necessary.  In 
every  case  the  reliability  and  the  computing  time  of  the 
classifier  is  strongly  dependent  upon  the  size  of  the  reference 
database.  Therefore  it  is  necessary  to  include  all  the  available 
sensor  data  (e.g.  sonar,  radar)  in  the  classification  process  to 
reduce  the  amount  of  models  in  the  database  as  much  as 
possible.  This  is  even  more  significant  for  the  neuronal  than  for 
the  correlative  classifier.  In  the  contrary  the  neuronal  classifier 
needs  an  considerable  smaller  amount  of  silhouettes  than  the 
correlation  classifier.  The  comparison  of  the  classification  error 
rates  is  difficult,  as  the  number  of  models  in  the  database  is  to 
different. 
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PAPER  No:  10 

DISCUSSOR’S  NAME:  N.J.  Farsaris 

COMMENT/QUESTION: 

This  seems  to  be  a  very  interesting  recognition  system.  What  is  the  maximum  range  of 
recognition/classification  obtained  using  that  system? 


AUTHOR/PRESENTER’S  REPLY: 

Due  to  the  focal  length  of  our  IR-system,  it  is  in  the  order  of  2-3  km  for  the  moment. 


DISCUSSOR’S  NAME:  P.  Verlinde 

COMMENT/QUESTION: 

Did  you  include  a  “rejection  option”  into  your  automatic  identification  scheme? 


AUTHOR/PRESENTER’S  REPLY: 

Up  to  this  moment  a  “rejection  option”  is  not  included.  The  classification  only  makes  a  proposal 
which  must  be  verified  by  the  operator. 

However,  a  “rejection  option”  can  be  added. 


DISCUSSOR’S  NAME:  G.S.  Brown 

COMMENT/QUESTION: 

Why  must  the  human  remain  in  the  decision  loop? 


AUTHOR  PRESENTER’S  REPLY: 

The  human  makes  the  definite  decision;  therefore  he  (or  she)  must  remain  in  the  loop! 
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PAPER  No:  10 

DISCUSSOR’S  NAME:  A.J.  van  der  Wal 

COMMENT/QUESTION: 

1 .  Why  the  preference  for  the  Kohonen  revised  network???? 

2.  Is  it  because  of  its  self-organisation  aspect? 

3.  Did  you  also  try  other  NN  architectures? 

AUTHOR/PRESENTER’S  REPLY: 

Multi-layer  perceptions  have  been  tried,  but  they  didn’t  perform  well.  Perhaps  because  we  didn’t 
have  enough  being  done.  Kohonen  images  seemed  to  perform  the  best  in  identification. 


11 


Advanced  Electro  Optic  Targeting  System 
Richard  W.  Jacobs/  Henry  S.  Lapp 
US  Air  Force,  Wright  Lab 
Targeting  Systems  Branch, 
Electro-optic  Technology  Division 
WL/AAJS,  Bldg  622, 3109  P  St. 

Wright  Patterson  AFB,  Ohio  45433-7700 


Abstract 

Future  Aircraft  targeting  systems  are 
integrating  sensor  inputs  for  automated 
target  detection,  location  and 
identification  with  the  goal  to  both  lower 
system  cost  and  increase  system 
performance.  This  paper  describes  a 
common  aperture  electro  optical  (EO) 
targeting  concept  for  an  integrated  air-to- 
ground  EO  targeting  system  using 
promising  developments  in  multispectral 
target  detection  and  in  laser  target 
identification.  The  integrated  capability 
allows  use  of  a  minimum  system 
aperture  while  increasing  recognition 
and  targeting  ranges  for  targeting  outside 
missile  threat  envelopes.  The  technical 
status  and  preliminary  test  results  of  the 
Wright  Laboratory  thrusts  to  develop 
both  laser  identification  and  thermal 
spectral  detection  capabilities  will  be 
discussed.  These  involve  developments 
and  demonstrations  of  multi-spectral 
detection  using  a  few  thermal  bands  and 
non-cooperative  target  identification 
using  a  burst  illumination  laser  with  eye- 
safe  wavelength  and  a  short  wave 
infrared  gated  camera.  These  individual 
concepts  will  be  mature  enough  to 
support  an  integration  effort  in  the 
1999/2000  timeframe.  The  cost, 
performance  and  level  of  automation 
supports  use  for  both  manned  and 


unmanned  air  vehicle  targeting  and 
reconnaissance  missions. 

1.  The  Need  for  an  Advanced  EO 

Targeting  System 

For  the  past  several  years  the  USAF 
has  been  interested  in  building  and 
demonstrating  extended  range  EO 
systems  that  provide  a  targeting 
capability  from  stand  off  ranges  outside 
the  missile  threat  envelope,  using  a 
smaller  aperture  compatible  with  lower 
aircraft  observability  concerns.  In  the 
early  1990’s  work  was  sponsored  to 
integrate  the  advantages  of  both  the 
FLIR  and  Laser  with  the  “FLASER”  or 
laser  plus  flir.  Flaser  was  designed  to 
demonstrate  detection  of  hot  targets  in 
the  open  with  the  new  advanced  flir 
detectors  and  coupled  with  a  burst 
illumination  laser  for  target  ID.  This 
work  was  down  scoped  due  to  funding 
problems  but  the  work  content  continued 
in  parts  under  other  programs.  The 
advantages  of  using  the  Laser  as  a  burst 
illuminator  of  potential  targets,  similar  to 
a  flash  lamp  in  photography,  and  then 
using  the  flir  to  detect  targets  in  the  open 
and  filter  out  the  clutter  has  shown  great 
potential.  Section  3  will  provide  more 
detail  with  the  discussion  of  the 
Enhanced  Recognition  and  Sensing 
Ladar  (ERASER)  program.  Addtionally 
it  is  clear  that  future  targeting  should  be 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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done  at  extended  ranges  whenever 
possible  such  that  the  ingressing  aircraft 
platform  can  “stand-off’  outside  the 
ranges  of  surface  to  air  missiles.  This 
requirement  also  translates  into  being 
able  to  recognize  and  identify  friend,  foe, 
or  neutral  forces  at  the  extended  ranges. 
EO  sensors  or  systems  offer  distinct 
advantages  for  recognition  with  high 
directionality  and  good  spatial 
resolution.  A  paper  by  Eismann  et  al 
showed  that  a  high  confidence  of  target 
detection  and  identification  can  be 
achieved  with  a  detection  probability  of 
0.9  and  a  false  alarm  rate  of  0.01  to 
0.001  per  square  kilometer  of  area 
searched.  The  current  state  of  the  art  in 
FLIR  systems  allows  for  good  capability 
for  targets  in  the  open  or  which  are 
partially  concealed  but  have  high  target 
to  background  contrast.  However,  for 
targets  in  camouflage,  concealment,  and 
deception  (CC&D),  current  capability  is 
poor.  The  other  concern  is  that  the  EO 
targeting  sensor  is  not  an  all  weather 
sensor  so  most  attack  aircraft  will  have 
radars  that  provide  the  capability  to 
detect  tactical  sized  targets  in  the  open  or 
moving.  The  radar  is  the  baseline 
targeting  sensor,  with  the  EO  system  as 
the  clearer  weather  capability  that 
provides  a  cost  effective  capability  to 
find  deep  hide  targets  and  to  guide 
precision  weapons.  This  motivates  us  to 
strive  to  build  a  targeting  system  in  the 
future  that  can  be  capable  in  these  more 
demanding  situations.  Whatever 
solutions  we  recommend  in  the  future 
must  be  additionally  cost  effective  or 
affordable.  One  aspect  to  achieve  that 
may  be  to  share  resources  by  integrating 
the  best  of  both  radar  and  EO  systems  to 
achieve  enhanced  recognition  and  ID. 


Alternatively,  it  may  not  be  necessary  to 
have  a  complete  EO  system  on  every 
offensive  aircraft  but  rather  to  share  the 
resources  between  a  fleet  of  them  or  to 
use  various  assets  between  manned 
fighters,  unmanned  aircraft,  or  Joint 
Stars  vehicles. 

2.  Standoff  Capability 

Recent  operational  requirements  are 
driving  us  toward  increased  range  ID  and 
targeting.  In  particular,  it  can  be  shown 
that  if  one  can  do  these  functions  at 
ranges  of  at  least  13  km,  one  can  stand 
off  and  survive  all  or  most  surface  to  air 
missile  threats.  The  objectives  of  our 
programs  are  to  extend  this  standoff 
range  as  far  as  possible,  but  recent 
developments  show  promise  of  doing 
this  at  about  20km  range.  One  of  the  key 
contributors  to  this  extended  range  is  the 
ERASER  program. 

3.  Enhanced  Recognition  and 
Sensing  LADAR 

The  ERASER  program  has  two 
objectives,  the  first  being  to  demonstrate 
air  to  ground  identification  of  ground 
targets  using  modified  eye  safe  laser 
designators  packageable  into  existing  flir 
targeting  systems.  This  will  provide 
future  pilots  with  positive,  timely,  and 
reliable  target  ID  and  Identification 
Friend  or  Foe  (IFF)  capability  as  an 
upgrade  to  existing  operational  flirs. 

The  second  objective  is  to  provide 
detailed  engineering  performance 
measurements  that  will  support  flir 
upgrade  decisions  and  performance 
estimations.  ERASER  will  also 
evaluate  the  capability  to  achieve  a 
limited  air-to-air  ID  capability  by  taking 
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advantage  of  laser  vibrometry  and 
microdoppler  developments  and  related 
laser  sources  being  developed.  Several 
key  points  about  the  ERASER  program 
make  it  unique.  The  identification  is 
done  using  a  short  laser  pulse  to 
illuminate  the  target  of  interest  for  an 
instant,  then,  a  range  gated  image 
intensified  Charge  Coupled  Device  IR 
camera  is  used  to  both  filter  out  the 
backscatter  and  receive  the  optical  pulse 
for  recognition.  This  offers  a  distinct 
advantage  over  scanning  systems  as  the 
single  pulse  approach  is  nearly  covert. 
This  serves  to  minimize  detection  of  the 
source.  The  ERASER  backscatter 
reduction  benefit  is  analogous  to  the 
effect  of  achieving  greater  visibility 
from  regular  headlights  over  high  beam 
lights  while  driving  an  automobile  in  the 
high  backscatter  conditions  of  a  snow  or 
sand  storm.  Additionally  ERASER 
will  make  use  of  the  same  laser  system 
for  recogition  as  other  laser  functions 
onboard  and  in  particular  the  same  laser 
source  as  the  laser  designator/ranger. 

The  intent  is  thus  to  effect  cost  savings 
and  prevent  proliferation  of  sensors, 
apertures,  and  functions.  An  advantage 
of  the  ERASER  approach  over  a  normal 
FLIRsystem  is  the  active  illumination  of 
the  target,  and,  a  decreased  dependence 
on  environmental  factors  and 
background  irradiance  by  using  an  active 
system  that  uses  target  reflectivity. 

Also,  ERASER  expects  to  use  eye  safe 
laser  wavelengths  (~1.5  um)  for 
operational  safety.  This  is  projected  to 
be  accomplished  through  either  Optical 
Parametric  Oscillator  (OPO),  or  a  Raman 
shifter,  to  a  1.06um  pump  laser.  Several 
approaches  to  this  wavelength  shifting 


include  a  potassium  titanyl  phosphate 
(KTP)  OPO,  or  an  erbium  doped  fiber 
pre-amplifier.  In  addition,  a  new  solid 
state  laser  designed  for  1 .5  micron  as 
fundamental  wavelength  may  be  feasible 
soon. 

The  ERASER  program  has  had  a  goal  to 
use  existing  reflective  common  aperture 
FLIR  designator  system  hardware  to 
affect  retrofit  and  affordability.  In 
addtition  ERASER  could  include  a 
degree  of  Automatic  Target  Recognition 
(ATR)  capability  although  it  will  also 
maintain  the  pilot  in  the  loop  prior  to 
weapon  release.  Ideally  ERASER  would 
be  used  to  provide  positive  target 
identification  (x,  y  TV  like  picture) 
along  a  “soda  straw”  for  the  pilot  prior  to 
launching  a  laser  guided  weapon. 
Presently  pilots  prefer  to  positively 
confirm  the  target  identity  prior  to  actual 
weapon  launch.  The  paper  by  Watson  et 
al  indicated  that  the  desired  long  range  of 
ERASER  should  be  achievable  with  a 
transmitted  energy  of  about  400  mJ  for 
the  eye  safe  wavelengths. 

Progress  on  ERASER  has  been 
encouraging  in  that  preliminary  tests  in 
1996  with  a  breadboard  system  showed 
good  results  to  detect  at  medium  range 
and  realistic  desert  conditions.  These 
tests  were  performed  at  Camp  Roberts  in 
California  during  National  Guard 
summer  exercises.  ERASER  performed 
well  at  ranges  of  approximately  3-10  km 
and  provided  eye  safe  2  Dimensional 
imagery.  The  brassboard  ERASER  was 
able  to  see  through  fog  oil  smoke 
intentionally  generated  around  a  tank  as 
target.  In  the  late  Summer  of  1997  an 
extended  test  is  planned  for  White  Sands 
Missile  Range  in  New  Mexico,  to 
evaluate  and  demonstrate  ERASER  at  up 
to  25km  range. 
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4.  Multispectral  sensing 

Multispectral  sensing  is  a  means  to 
exploit  the  spectral  information  of  a 
potential  target  by  using  several 
wavelengths  of  interest.  Multispectral 
differs  from  hyperspectral  in  that  in  the 
former  2-3  key  wavelengths  are  used 
rather  than  dozens  with  the  latter,  thus 
relieving  the  hardware  and  processing 
demands  of  the  system.  Desert  Storm 
showed  the  need  to  automatically  detect 
and  identify  military  targets  with  very 
low  false  alarms  as  well  as  to  locate 
targets  with  sufficient  accuracy  to 
perform  precision  strike.  Additionally 
targets  with  Camouflage,  Concealment, 
&  Deception  (CC&D)  or  in  deep  hide 
make  detection  more  difficult. 
Multispectral  systems  offer  the  capability 
to  perform  limited  area  search  from  a 
passive  sensor  from  low  to  high 
altitudes.  Allthough  limited  by  weather, 
thermal  multispectral  offers  a  day  or 
night  passive  detection  capability. 
Multispectral  sensing  requires 
significantly  less  spatial  resolution  so 
systems  can  be  designed  that  image 
wider  areas  than  single  band  spatial 
sensors. 

By  using  multispectral  detection  in 
selected  wavelength  bands  germane  to 
the  targets/backgrounds/atmospheres  of 
interest,  e.g.,  the  wavelengths 
characteristic  of  vehicle  paints, 
camouflage  nets,  etc.,  it  is  possible  to 
exploit  these  characteristics  and  augment 
the  detection  capabilities  of  passive  IR 
systems.  Multispectral  is  used  for  target 
detection  enhancements  by  a  careful 
reduction  in  the  background  clutter. 

This  is  done  by  choosing  band 
combinations  for  which  the  clutter  data 


is  highly  correlated  between  bands. 
When  done  correctly,  the  suppressed 
clutter  level  is  less  sensitive  to  typical 
diurnal  variations  in  environment  of 
single  band  clutter.  The  few  bands  are 
selected  to  take  advantage  of  the 
correlation  of  the  background.  Ideally 
two  bands  could  be  chosen  that  have  all 
the  background  pixels  very  correlated 
(need  definition  of  correlation)  and  the 
target  pixels  with  lots  of  color  relative  to 
the  background.  A  simple  subtraction  of 
the  images  would  then  leave  only  target 
pixels.  Many  years  of  tower 
experiments  with  both  imaging  and  non¬ 
imaging  thermal  hyperspectral 
instruments  against  various  backgrounds 
at  various  times  of  the  day/night  and 
seasons  have  shown  that  backgrounds 
consistently  do  exhibit  a  high  degree  of 
correlation.  The  general  case  however 
requires  more  than  two  bands  to  achieve 
high  probabilities  of  target  detection  and 
more  importantly  low  false  alarms. 

Results  so  far  have  shown  the 
potential  for  multispectral  systems  using 
both  tower  and  airborne  imaging 
systems.  In  parallel,  new  detection 
algorithms  have  been  developed  for 
optimum  performance  and  minimum 
false  alarms.  The  paper  by  Eismann  et 
al  showed  that  detection  at  greater  than 
15  km  range  is  feasible  for  both  Mid 
Wave  and  Long  Wave  (MW  and  LW)  IR 
systems.  The  thermal  multispectral 
program  is  progressing  well  in  the  U.S. 
with  tower  tests  of  both  imaging  and 
non-imaging  sensors  completed  and 
flight  tests  planned  starting  in  the  end  of 
1998.  Programs  are  planned  or  in  place 
to  mature  multispectral  techniques  so 
that  they  should  be  available  for  an 
advanced  eo  targeting  system  integration 
effort  after  the  year  2000. 
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5.  Shared  Resources 

Future  weapons  systems’  costs  need 
to  be  significantly  lower.  Lower  costs 
can  be  achieved  by  integrating  functions, 
reducing  the  cost  of  components,  sharing 
subsystem  components,  increasing 
performance  or  increasing  aircraft 
survivability.  The  proposed  integrated 
targeting  concept  impacts  the  final 
system  cost  using  all  of  the  mentioned 
cost  reduction  options.  The  stabilization 
requirements  for  an  advanced  eo 
targeting  system  like  this  should  be  less 
demanding  and  lower  cost  because  the 
pixel  size  previously  required  for  a  flir  to 
do  detection  is  now  much  larger  since 
we  are  doing  spectral  detection  not 
spatial  detection.  The  pixel  size  could 
grow  from  a  foot  or  two  to  a  meter  or 
two.  The  short  pulse  on  the  laser 
illuminator  also  minimizes  the  need  for 
expensive  stabilization  since  motion  is 
stopped  for  the  duration  of  the 
illumination  pulse.  The  use  of  larger 
pixels  for  detection  and  the  use  of  the 
shortest  wavelength  possible  while  still 
maintaining  eye  safe  laser  illumination 
allows  minimum  apertures  and  aircraft 
windows  to  be  employed  -  again  a  cost 
saving  over  current  systems.  The 
integration  of  the  radar  and  EO 
information  provides  another  source  of 
cost  savings.  It  is  possible  to  detect  a 
potential  target  with  a  radar  or  other 
cueing  sensor  that  hands  off  the  target 
detection/location  to  a  attack  aircraft 
only  equipped  with  the  EO  system.  It  is 
also  possible  to  use  the  EO  equipped 
aircraft  for  detection/ID/designation 
and  use  a  third  aircraft  as  the  carrier  of 
laser  guided  or  other  precision  weapons. 
These  targeting  concepts  could  mean  that 


not  every  airplane  in  a  future  wing  has 
both  radar  and  eo  but  rather  a  team 
approach  is  used.  Or,  other  assets  such 
as  unmanned  aerial  vehicles  could  be 
used  to  cue  the  attack  platform.  These 
concepts  should  provide  for  cost 
effective  identification,  recognition  and 
targeting. 

6.  Risk  Assessment 

The  chief  risks  associated  with  an 
advanced  EO  targeting  system  built  on 
the  preceding  technologies  and  concepts 
are  now  outlined.  ERASER  is  built  up 
from  largely  existing  laser  technologies 
and  the  new  generation  of  reflective 
aperture  flirs  The  primary  laser  is  either 
a  currently  available  1.06  laser 
designator  which  is  shifted  in 
wavelength  to  1.54  or  1.56  u,  or, 
emerging  solid  state  lasers  that  offer 
more  power  and  packaging  potential 
while  still  providing  the  multifunctions 
of  1 .06  u  ranging,  designation  and  at 
1.54  or  1.56  u  target  illumination.  There 
is  low  risk  in  using  the  available 
designator  laser  although  medium  risk 
for  the  wavelength  shifting  or  adding 
more  power  needed  to  achieve  longer 
ranges.  The  Infrared  gated  camera 
system  with  ERASER  is  judged  low  risk. 
ERASER  itself  will  be  demonstrated 
using  largely  retrofitted  aircraft  parts 
(e.g.,  apertures,  optics)  in  the  next  three 
years  which  serves  as  risk  mitigation  for 
a  system  program.  Thus  ERASER  is 
medium  to  low  risk.  Multispectral 
sensing  is  also  being  actively  pursued 
with  programs  to  develop  and 
demonstrate  both  day  only  reflective 
multispectral  and  day/night  thermal 
multispectral  capabilities  in  brassboard 
applications.  The  thermal  instrument  is 
planned  to  fly  at  both  low  and  high 


altitudes  to  gather  sufficient  background, 
target  and  atmospheric  data  to  support 
both  the  system  utility  assessment  as 
well  as  system  design  trades.  By  2000 
the  thermal  spectral  technology  should 
be  mature  enough  to  be  considered  as  a 
candidate  for  integration  into  a  targeting 
system.  Day  only  reflective 
multi  spectra]  technology  is  judged 
medium  to  low  risk,  but  for  overall  (day 
and  night)  use  the  thermal  multispectral 
capability  is  still  high  risk.  The 
Automatic  Target  Recognition  role 
remains  to  be  mitigated  and  we  expect  it 
to  require  additional  development  for  the 
ERASER  and  Mulitspectral 
technologies.  It  is  judged  medium  to 
high  risk.  Finally,  the  integration  and 
demonstration  of  the  whole  system  is 
judged  medium  to  high  risk. 

7.  Application  &  Summary 

We  believe  that  the  promising 
developments  outlined  above  should 
lead  to  an  advanced  EO  targeting  system 
for  a  host  of  applications.  In  the  USAF 
we  believe  that  our  plans  will  be  of 
primary  interest  to  Air  Combat 
Command  in  support  of  such  aircraft  as 
the  Joint  Strike  Fighter  as  well  as  various 
upgrades  to  existing  fighter  platforms 
such  as  FI  5  or  FI 8.  Part  of  the  Air 
Force  vision  for  future  combat 
identification  is  for  the  combination  of 
on  and  off  board  sensors  used  in  a 
system  of  systems  approach  with 
interoperable  data  links.  Additionally  the 
system  concepts  outlined  above  should 
be  of  interest  for  use  in  Unmanned 
Aerial  Vehicles.  In  summary  we  believe 
we  are  working  toward  the  next 
generation  eo  targeting  system  and  that 


in  particular  the  efforts  in  ERASER  and 
Multispectral  sensing  are  key  mitigators 
of  the  risk.  We  believe  that  the  next 
generation,  advanced  EO  targeting 
system  can  be  built  and  demonstrated 
using  these  key  technologies  which 
should  be  available  post  2000  time 
frame.  We  look  forward  to  integrating 
these  into  an  advanced  system  and 
demonstrating  them  for  the  purposes 
mentioned. 
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Current  research  on  Raman-shifted  Nd:  YAG  and  Er :  Glass  at/=  1.53jim. 
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SOMMAIRE 

Le  propos  de  cet  expose  est  de  decrire  la  metho- 
dologie  qui  a  ete  developpee  pour  revaluation  des 
poursuites  multicapteur  au  sein  de  I’Agence  C3  de 
I’OTAN.  Cette  methodologie  a  originellement  ete 
developpee  dans  le  but  de  quantifier  les  exigences 
de  performances  en  termes  applicables  a 
I’etablissement  de  la  situation  aerienne  renseignee 
(, Recognized  Air  Picture  RAP)  dans  le  cadre  de  la 
redaction  des  specifications  du  futur  systeme  de 
commandement  et  de  controle  des  operations 
aeriennes  (Air  Command  and  Control  System, 
ACCS)  de  I’OTAN.  Un  certain  nombre  de  mesures 
de  performances  (MOPs)  ont  ainsi  ete  definies.  Un 
processus  de  segmentation  des  pistes  et  des 
trajectoires  des  cibles  poursuivies  s’impose  comme 
condition  prealable  au  calcul  d’une  partie  de  ces 
mesures:  la  description  de  cette  segmen-tation 
s’inscrit  egalement  dans  la  methodologie 
developpee,  tout  comme  la  mise  en  oeuvre  de 
differents  filtres  permettant  une  etude  ciblee  des 
performances  de  poursuite. 

La  description  fonctionnelle  de  I’architecture  de  ce 
systeme  devaluation  detaille  les  problemes 
temporels  et  spatiaux  ainsi  que  la  maniere  dont  ils 
ont  ete  traites.  De  meme,  differentes  approches 
sont  etudiees:  la  simulation,  I’utilisation  de  donnees 
reelles  et  le  calcul  analytique. 

LISTE  DES  ACRONYMES 

Les  acronymes  suivants  sont  utilises  a  plusieurs 
reprises  dans  ce  document. 

ACCS  Air  Command  &  Control  System 

Systeme  de  commandement  et  de 
controle  des  operations  aeriennes 
MOP  Measure  of  Performance 

Mesure  de  performance 
RAP  Recognized  Air  Picture 

Situation  aerienne  renseignee 
SFP  Sensor  Fusion  Post 

Cabine  de  fusion  de  donnees 


IDCP  Identification  Data  Combining  Process 
Processus  d’identification  par 
combinaisons  des  donnees. 

INTRODUCTION 

L’approche  retenue  lors  de  I’elaboration  de  cette 
methode  etait  guidee  par  le  desir  de  maintenir  des 
liens  etroits  entre  les  exigences  operationnelles 
des  utilisateurs  et  les  diverses  mesures  analyti- 
ques  de  performance.  Chaque  mesure  peut  ainsi 
etre  ponderee  selon  1’importance  qu’elle  requiert 
aupres  des  differents  utilisateurs:  en  effet,  les 
besoins  des  “usages”  de  la  situation  aerienne 
different  considerablement,  et  il  semblait  important 
de  ne  pas  chercher  a  attribuer  une  performance 
globale  mais  plutot  de  presenter  les  resultats  en 
fonction  des  differents  profils  d’utilisateurs. 

Cet  expose  se  concentre  sur  revaluation  des 
performances  de  poursuite  multicapteur  dites  a 
“fusion  de  plots”  plutot  qu’a  “fusion  de  piste”,  mais 
la  methode  s’etend  aisement  a  tous  les  types  de 
fusion  de  donnees.  Les  exemples  choisis  concer- 
nent  I’etage  de  fusion  des  detections  fournies  par 
les  differents  capteurs  du  theatre  d’operation  qui 
est  materialise  par  le  “Sensor  Fusion  Post  (SFP)” 
dans  I’architecture  de  I’ACCS. 

METHODOLOGIE 

Trois  approches  sont  offertes  pour  revaluation  de 
performances;  la  simulation,  I’utilisation  de  don¬ 
nees  reelles  et  le  calcul  analytique.  Elies  corres¬ 
pondent  a  des  besoins  differents  et  peuvent  etre 
complementaires.  Independamment  de  I’approche 
choisie,  la  premiere  etape  consiste  a  etablir  un 
environnement  operationnel  dans  lequel  des 
exigences  de  qualite  et  de  detection  peuvent  etre 
definies. 

2.1.  Exigences  Op6rationnelles 
La  plupart  des  utilisateurs  ne  sont  pas  preoccupes 
par  les  performances  techniques  de  la  fonction  qui 
elabore  la  situation  aerienne.  Ils  ont  une  notion 
confuse  concernant  la  stabilite  des  pistes  qu’ils 
regoivent,  le  temps  de  reaction  apres  une 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
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manoeuvre,  la  creation  de  fausses  pistes  dans  les 
zones  de  fouillis,  etc.  L’expression  d’exigences 
operationnelles  est  d’autant  plus  difficile  que  les 
utilisateurs  integrent  souvent  les  problemes 
auxquels  ils  sont  confrontes  avec  un  systeme 
donne*1).  Les  specifications  d’un  systeme  reposent 
ainsi  parfois  sur  la  correction  de  problemes 
rencontres  avec  le  systeme  precedent  et  non  pas  a 
I’exigence  operationnelle  originale.  L’analyse  des 
specifications  doit  ainsi  tenir  compte  du  bais  naturel 
qu’ont  les  operateurs  vis  a  vis  du  systeme  en 
place. 

La  Figure  2  illustre  le  concept  de  revaluation  de 
performance  d’un  systeme  de  poursuite 
multicapteur.  La  reference  est  constitute  par  le 
scenario  tel  qu’il  a  ete  genere  et  “joue”  pour 
produire  les  detections  des  capteurs.  L’objectif  du 
concept  et  d’analyser  quelles  sont  les 
performances  affectees  par  quels  facteurs,  afin  de 
mieux  cerner  la  sensibilite  du  systeme  a  ces 
differents  parametres.  Au  cours  du  processus 
d’elaboration  de  la  situation  aerienne,  les  capteurs 
produisent  d’abord  leur  vision  de  I’environnement, 
que  I’on  pourrait  appeler  la  situation  mesuree.  Puis 
le  pistage  produit  par  filtrage  des  mesures  une 
situation  estimee  (dont  la  qualite  est  en  principe 
superieure  a  celle  de  la  situation  mesuree).  Enfin, 
une  situation  predite  est  calculee  pour  etre 
visualisee  par  les  operateurs.  L’un  des  objectifs  de 
revaluation  est  de  quantifier  les  facteurs  qui  limitent 
ou  affectent  sensiblement  les  performances  du 
systeme.  Les  methodes  mises  en  oeuvre 
travaillent  soit  par  comparaison  entre  les  entrees  et 
les  sorties  (situation  predite  par  rapport  a  la 
reference  par  exemple),  soit  par  analyse  des 
processus  et  des  modeles  impliques. 

2.2.  Methode  par  Simulation 

La  Figure  1  illustre  la  methode  par  simulation  qui 
traite  la  poursuite  multicapteur  comme  une  “boite 
noire”,  recevant  les  detections  des  capteurs  en 
entree  et  delivrant  des  pistes  en  sortie. 


1  Certains  operateurs  disposent  de  la  visualisation  des 
plots  synthetiques  et  peuvent  observer  des  manoeuvres 
avant  que  la  piste  ne  rende  compte  de  1’evolution  des 
cibles.  S’ils  sont  en  liaison  radio  avec  les  pilotes,  ils 
peuvent  savoir  qu’un  chasseur  manoeuvre  avant  meme 
que  le  systeme  ne  le  detecte. 
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Figure  1:  Methode  devaluation  par  simulation 


2.2.1.  Scenario 

L’  environnement  de  simulation  ne  requiert  pas  un 
realisme  tactique  (representation  exacte  d’une 
menace),  mais  doit  comprendre  un  certain  nombre 
d’elements  caracteristiques  qui  constituent  les 
points  d’interet  d’un  ensemble  d’utilisateurs.  Les 
exigences  operationnelles  sont  ensuite  reliees  a 
une  ou  plusieurs  mesures  de  performance.  Ces 
references  croisees  sont  etablies  par  i’etude  de 
chaque  caracteristique  de  la  situation  aerienne  et 
par  selection  des  performances  qui  traduisent  la 
qualite  de  cette  caracteristique.  L’evaluation  de  la 
performance  finale  sera  conduite  en  fonction  de 
ces  references  croisees  apres  avoir  applique  les 
ponderations  et  filtrages  definis  par  les  utilisateurs. 

De  maniere  generale,  un  scenario  se  caracterise 
par  trois  composantes: 

•  Les  moyens  de  detection,  a  savoir  le  nombre,  le 
type  et  les  caracteristiques  des  capteurs,  leur 
position  dans  I’espace,  le  recouvrement  des 
couvertures  electromagnetiques,  le  terrain,  les 
moyens  de  communications  utilises  pour 
acheminer  les  informations  vers  le  centre  de 
fusion, 

•  La  configuration  aerienne  des  objets  aeriens 
(nombre  d’aeronefs,  type,  vitesse,  altitude, 
manoeuvres,  signature,  emission  electro- 
magnetique,  separation  des  appareils,  etc.), 

•  L’environnement  electronique  et  meteorolo- 
gique,  tel  que  la  temperature,  les  precipitations, 
le  vent,  le  brouillage,  etc. 

Ce  que  Ton  appelle  communement  un  scenario 
consiste  en  fait  en  une  famille  de  cas  voisins  dont 
certains  parametres  de  ces  trois  composantes 
varient  (memes  moyens  de  detection,  memes 
aeronefs,  conditions  climatiques  differentes  par 
exemple).  Ainsi,  on  pourra  etudier  les  variations  de 
performance  en  fonction  de  1'evolution  de  certains 
parametres. 


2.2.2.  Monte-Carlo 

Le  processus  complet  d’elaboration  de  la  situation 
aerienne  est  tres  complexe  parce  qu’il  englobe  des 
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sous-processus  notoirement  differents,  tels  que  la 
detection,  la  transmission  des  donnees,  le  filtrage 
numerique,  et  la  visualisation  des  pistes.  Chaque 
niveau  est  une  source  d’erreur  potentielle  de  telle 
sorte  que  I’analyse  du  processus  complet  n’est  pas 
deterministe.  Devant  une  telle  complexite,  il  est 
necessaire  de  recourir  a  des  techniques  de  rejeu 
d’un  meme  scenario,  en  appliquant  des  conditions 
initiates  differentes,  ce  qui  permet  d’accroTtre  le 
degres  de  confiance  que  Ton  peut  attribuer  aux 
resultats.  Les  techniques  de  rejeu  permettent  de 
rassembler  un  nombre  plus  important 
d’echantillons  d’analyse  et  ainsi  d’estimer,  en  plus 
des  valeurs  de  performance,  la  dispersion  des 


•  II  n’existe  pas  de  donnees  de  reference  (c’est  a 
dire  les  trajectoires  du  scenario) 

•  II  n’est  pas  aise  d’effectuer  des  mesures 
statistiques  (le  volume  des  donnees  reste  limite 
et  le  rejeu  reste  tres  limite). 

II  est  possible  de  pallier  plus  ou  moins  la  premiere 
contrainte  en  utilisant  des  aeronefs  equipes  de 
moyens  de  localisation  tres  precis,  tels  qu’une 
balise  GPS  ( Global  Positioning  System) 
embarquee  ou  une  centrale  inertielle  precise.  Ces 
equipements  fournissent  des  informations  qui 
peuvent  etre  utilisees  pour  recalculer  les 
trajectoires  a  posteriori. 


ibles 

-  Numbre  d'a6ronefs 

-  Cin6matique 

j-  Caract6ristiques 


Capteurs 

Numbre  de  capteurs 
|-  Cconfiguration 
Caract6ristiques 


j  Environnc 

L  Terrain 
j-  Interferences, 


Evaluation  Comparative 


Figure  2:  Concept  de  revaluation  de  performances  de  poursuite 


mesures. 

2.2.3.  Utilisation  de  cette  methode 

Cette  methode  permet  ainsi  non  seulement  de 
comparer  les  performances  de  differents  types  de 
poursuites  (voire  de  differents  reglages),  mais 
encore  de  mesurer  a  quel  point  les  exigences 
operationnelles  sont  satisfaites  et  comment  la 
poursuite  repond  aux  besoins  des  utilisateurs.  Elle 
autorise  egalement  I’analyse  de  phenomenes  aux 
limites  et  de  comportements  extremes,  qui  -  bien 
que  non  realistes  -  permet  de  mieux  comprendre  le 
comportement  de  I’ensemble  materiel  et  logiciel 
que  forme  un  systeme  de  poursuite  multicapteur. 

2.3.  Execution  de  donnees  rtelles 

Les  tests  et  verifications  des  algorithmes  de 
poursuites  par  des  moyens  de  simulation  est  une 
etape  necessaire,  cependant  il  est  egalement  utile 
de  valider  les  performances  vis  a  vis  d’un 
environnement  reel.  La  methode  par  simulation 
peut  s’appliquer  a  des  donnees  reelles  avec  les 
deux  contraintes  majeures  suivantes: 


Si  Ton  dispose  d’enregistreurs  synchronises 
capable  de  stacker  simultanement  les  detections 
des  capteurs,  puis  de  rejouer  ces  donnees,  en  les 
mixant  eventuellement  avec  des  donnees  simules 
(fouillis  par  exemple),  on  peut  augmenter  les 
echantillons  d’analyse. 

2.4.  Methode  Analvtique 

La  mise  en  place  d’une  simulation  de  poursuite 
multicapteur,  avec  rejeux,  peut  sembler  lourde  et 
couteuse  lorsque  Ton  cherche  a  obtenir  des 
resultats  rapidement  ou  a  analyser  un  probleme 
particular.  La  prediction  des  performances  par  une 
methode  anaiytique  peut  alors  s’averer  souhaitable 
pour  estimer  les  limites  d’un  ensemble  donne  de 
capteurs  et  d’aeronefs,  et  pour  identifier  les 
parametres  preponderants.  Ces  limites  ont 
I’avantage  d’etre  generiques  dans  la  mesure  ou 
elles  ne  dependent  pas  d’un  scenario  particuiier 
(avec  les  autres  methode,  un  doute  persiste 
toujours  quant  a  la  pertinence  du  scenario  ou  des 
donnees  enregistrees).  Les  performance  obtenues 
permettent  de  delimiter  le  perimetre  des  possibility 
d’un  algorithme  donne  ou  d’une  configuration  de 
capteurs  specifiques.  Bien  sur,  il  n’y  a  aucune 


13-4 


garantie  que  les  performances  calculees  puissent 
etre  effectivement  obtenues  par  un  systeme  reel. 
Par  contre,  si  les  performances  estimees  ne 
satisfont  pas  les  exigences,  il  est  certain  que  le 
systeme  ne  sera  pas  conforme,  qu’il  y  a  un  risque 
important,  et  que  des  solutions  alternatives  doivent 
etre  envisagees. 

2.4.1.  Difficulte  de  la  methode 

devaluation  des  performances  d’un  systeme  de 
surveillance  est  une  tache  difficile  parce  qu’elle 
comprend  des  elements  conflictuels.  Un  systeme 
de  poursuite  est  un  systeme  stochastique  hybride 
qui  traite  deux  types  d’inconnues: 

•  Discretes  pour  ce  qui  concerne  la  detection 
(faux  plots,  fouillis):  la  presence  ou  I’absence 
d’un  echo,  de  meme  que  I’apparition  d’un  faux 
plot  peuvent  changer  radicalement  la  creation  et 
la  mise  a  jour  d’une  piste. 

•  Continues  quant  aux  erreurs  dynamiques  et  aux 
bruits  de  mesure:  les  covariances  d’erreur  sont 
remarquablement  traitees  par  les  algorithmes 
de  filtrage  numerique. 

2.4.2.  Champ  d’application 

L’etude  analytique  supportee  par  une  modelisation 
precise  du  systeme  et  des  traitements  mis  en 
oeuvre  est  souvent  necessaire  prealablement  a 
(’investigation  d’un  nouveau  concept. 

3,_  MESURES  DE  PERFORMANCE 

II  n’est  pas  possible  de  selectionner  seulement  une 
ou  deux  mesures  pour  evaluer  les  performances 
d’une  poursuite.  Les  criteres  d’exigence  se 
traduisent  generalement  au  travers  de  plusieurs 
MOPs.  Malgre  I’interdependance  des  mesures,  il 
est  souhaitable  de  definir  des  MOPs  qui  rendent 
compte  des  performances  de  certaines  sous- 
fonctions  comme  par  exemple  la  creation 
automatique  de  pistes  ou  la  correlation.  II  est 
important  de  mesurer  le  taux  de  bonnes  ou 
mauvaises  associations,  mais  il  ne  faut  pas  que  de 
les  erreurs  dissociations  condamnent  I’analyse  de 
la  precision  en  position  par  exemple  (voir  le 
paragraphe  3.1  sur  la  segmentation). 


Mesure 

definition 

Couverture 

Pourcentage  moyen  de  cibles 
repr6sentdes  effectivement  par  un 
piste  parmi  I'ensemble  des  cibles 
sur  chaque  intervalle  de  temps 

Vraisemblance 

Pourcentage  moyen  entre  le 
nombre  de  pistes  representant  une 
cible  rSelle  et  le  nombre  total  de 
pistes 

Delai  d’initialisation 

Delai  moyen  entre  la  premiere 
detection  d’une  cible  et  la 
confirmation  de  la  piste  associee. 

Taux  de 

rafraTchissement 

Periode  moyenne  de  mise  y  jour 
des  pistes. 

Precision  en  position 

Moyenne  des  diffyrences  entre  la 
position  des  cibles  et  de  la  piste  qui 
leur  est  associee. 

Precision  en  altitude 

Moyenne  des  differences  entre 
(’altitude  des  cibles  et  de  la  piste  qui 
leur  est  associye. 

Precision  en  cap 

Moyenne  des  diffyrences  entre  le 
cap  des  cibles  et  de  la  piste  qui  leur 
est  associee. 

Precision  en  vitesse 

Moyenne  des  diffyrences  entre  la 
vitesse  des  cibles  et  de  la  piste  qui 
leur  est  associee. 

Continuity 

Pourcentage  du  temps  pendant 
lequel  une  cible  est  representee 
continuement  par  la  meme  piste  au 
long  de  son  existence. 

Stability 

Moyenne  des  variations  de  cap 
dans  le  temps 

Taux  de  fausses  pistes 

Moyenne  du  nombre  de  fausses 
piste  crdes  et  maintenues  par  le 
systeme. 

Table  1:  Liste  de  mesures  de  performance 


L’une  des  gageures  est  de  limiter  le  nombre  des 
MOPs  pour  conserver  une  comprehension  globale 
et  intuitive  de  I’analyse.  Un  jeu  de  15  a  20  MOPs 
permet  une  granularity  d’introspection  acceptable. 
La  Table  1  illustre  les  MOPs  les  plus  couramment 
utilisees. 

Cette  liste  s’applique  essentiellement  aux 
performances  de  poursuite.  La  methodologie 
generate  peut  etre  appliquee  a  I’examen  de 
processus  plus  fin  tels  que  la  detection,  la  tenue 
d’une  piste  individuellement,  les  communications, 
ou  la  visualisation  de  la  situation  aerienne.  Des 
mesures  de  performances  similaires  peuvent  etre 
analysees  sous  differents  angles:  par  exemple,  le 
delai  d’initialisation  peut  etre  mesure  sous  un  angle 
operationnel  (delai  entre  la  penetration  d’un 
aeronef  dans  une  zone  de  defense  et  I’affichage  de 
la  piste  correspondante  sur  I’ecran  d’un  operateur) 
ou  technique  (delai  entre  I’instant  de  detection 
d’une  cible  et  la  creation  de  la  piste  qui  la 
represente  dans  la  table  des  pistes). 

3.1.  Segmentation 

Avant  d’analyser  une  piste,  il  est  necessaire  de  la 
decomposer  en  segments,  chaque  segment 
representant  une  partie  de  la  piste  pendant  laquelle 
celle-ci  represente  le  meme  aeronef.  Ce  pre- 
traitement  des  pistes  et  des  trajectoires  est 
indispensable  a  I’etude  objective  des  echanges 
piste/cible,  du  calcul  des  fausses  pistes  et  de 
I’estimation  des  precisions  en  position,  vitesse,  cap 
et  altitude. 
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Figure  3:  Environnement  du  Laboratoire  ASEP 


3.2.  Selection  des  Schantillons  par  filtrage 

II  est  clair  que  differents  utilisateurs  ont  des 
priorites  dissemblables  concernant  la  qualite  des 
pistes.  Ces  priorites  peuvent  varier  en  fonction  de 
parametres  tels  que  le  cap,  I’altitude,  la  vitesse  ou 
I’identite  des  pistes.  Par  exemple,  I’operateur  d’un 
systeme  de  missiles  sol-air  considerera  les  pistes 
rapides  et  approchantes  comme  les  pistes  les  plus 
importantes  en  comparaison  des  pistes  qui 
s’eloignent  de  sa  zone  de  responsabilite.:  ainsi,  il 
pourra  avoir  des  exigences  de  performance 
differentes  pour  ces  pistes.  Des  profiles  de  filtrage 
ont  done  ete  mis  en  oeuvre  pour  permettre  une 
selection  des  elements  a  prendre  en  compte  pour 
revaluation  de  performances  selon  la  position 
geographique  des  pistes,  leur  cap,  vitesse,  altitude, 
identite,  etc. 


3.3. 


Ponderation 


Les  mesures  de  performance  individuelles  sont 
calculees  pour  chaque  mise  a  jour  de  piste,  puis 
sont  globalisee  en  utilisant  une  moyenne  simple 
plutot  qu’une  moyenne  quadratique.  Cette  maniere 
de  proceder  est  plus  representative,  principalement 
pour  les  scenarios  comprenant  des  cibles  tres 
manoeuvrantes.  L’utilisation  d’une  moyenne 
quadratique  tend  a  rendre  trop  preponderants  les 
mauvais  resultats  obtenus  pendant  les  virages  et  a 
fausser  le  resultat  global.  En  effet,  la  brievete  des 
manoeuvres  (du  moins  dans  un  environnement 


militaire)  conduit  a  considerer  celles-ci  comme  une 
perturbation  apres  laquelle  la  piste  doit  rapidement 
reconverger  vers  la  cible  (la  precision  de  la  piste 
pendant  la  manoeuvre  n’a  pas  d’interet 
operationnel). 

Afin  d’eviter  que  quelques  mesures  particu- 
lierement  mauvaises  ne  viennent  denaturer  le 
resultat  global,  une  pre-selection  est 
automatiquement  appliquee  avant  d’effectuer  la 
ponderation.  Habituellement,  5%  a  10%  des  plus 
mauvaises  mesures  sont  rejetees  du  calcul  de  la 
moyenne  de  chaque  cycle.  Cette  elimination 
reflete  la  realite  dans  la  mesure  ou  les  operateurs 
concentrent  justement  leur  attention  sur  les  parties 
difficiles  (virages,  formations  serrees,  etc.)  et 
peuvent  agir  manuellement  pendant  ces  instants 
critiques. 

4.  LABORATOIRE  ASEP 

La  Division  commandement,  controle  et  capteurs 
de  I’Agence  C3  de  I’OTAN  (NC3A)  a  developpe 
une  plate-forme  ASEP  (Air  Surveillance  Exploratory 
Prototype).  Ce  laboratoire  dispose  d’un 
environnement  integre  permettant  I’experimentation 
des  fonctions  de  surveillance  et  d’identification 
avec  des  donnees  reelles  ou  simulees.  Des  lignes 
de  communication  acheminent  les  detections 
temps-reel  en  provenance  de  15  capteurs  civils  ou 
militaires,  les  pistes  delivrees  par  les  moyens 
d’alerte  aeroportes  (NAEW),  les  situations 
aeriennes  et  plans  de  vols  de  deux  centres  de 
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controle  (CRC)  et  du  centre  d’Eurocontrol  installe  a 
Maastricht  comme  le  montre  la  Figure  3. 

Les  moyens  de  simulation  comprennent  le 
generateur  de  scenario  SPACE  ( Scenario 
Preparation  and  Coverage  Evaluation)  et  le  modele 
MOCCA  (Model  of  Command  and  Control  of  Air 
Defence  Systems).  Enfin,  les  logiciels  MUTANT 
(Multi-Sensor  Tracking  Analysis  Tool)  et  SMART 
(Segmentation  and  MST  Analysis  and  Reporting 
Tool)  de  calcul  des  performances  et  I’outil  de 
visualisation  TSDS  (Track  and  Sensor  Display 
System)  completed  la  panoplie  des  outils 
informatiques  utilises  pour  revaluation  des 
performances  de  poursuite  qui  composent  le  projet 
RAPACE  (RAP  Analysis  &  Comparative  Evaluation) 
illustre  par  la  Figure  4. 

L’outil  de  generation  de  scenario  SPACE  permet 
de  definir  le  reseau  de  capteurs  en  s’appuyant  sur 
I’ensemble  des  couvertures  electromagnetiques  qui 
peuvent  etre  calculees  et  visualisees,  pour 
n’importe  quelle  tranche  d’altitude,  en  tenant 
compte  de  la  nature  du  terrain  (zone  urbaine, 
forets,  desert,  mer,  etc.)  et  du  relief.  Les 
trajectoires  des  aeronefs  peuvent  etre  ainsi  crees 
et  modifiees  en  ayant  connaissance  des  capacites 
de  detection  du  systeme. 


Figure  4:  Projet  RAPACE 

Les  scenarios  crees  par  SPACE  sont  deroules  par 
i’outil  MOCCA  qui  integre  des  modeles  fins  pour  un 
grand  nombre  de  capteurs.  Ces  modeles  simulent 
precisement  les  performances  de  capteurs  a 
impulsion,  impulsion  Doppler,  forme  d’onde 
continue,  2D  et  3D,  couvrant  I’ensemble  des 
caracteristiques  des  radars  primaires  et 
secondaires,  fixes  et  aeroportes,  actifs  et  passifs, 
susceptible  d’etre  reellement  utilises. 

L’outil  MOCCA  integre  egalement  un  modele  de 
poursuite  multicapteur.  Les  algorithmes  mis  en 
oeuvre  sont  equivalents  a  ceux  utilises  dans  les 
systemes  modernes  de  controle  aerien  civil  ou 
militaire. 

Les  fichiers  de  detection  et  ceux  des  pistes  sont 
conserves  dans  une  base  de  donnee  accessible 
par  I’ensemble  des  outils.  Ms  peuvent  egalement 


etre  enregistres  sur  des  fichiers  dans  un  format 
defini  par  i’Agence.  Cette  possibility  autorise 
I’execution  de  scenarios  reels  ou  simules  sur  des 
plate-formes  differentes,  par  exemple  des  bancs  de 
test  industriels,  des  systemes  en  cours 
d’implementation  ou  des  systemes  existants.  Les 
performances  peuvent  ainsi  etre  evaluees  a 
I’Agence  ulterieurement. 

L’usage  d’une  base  de  donnees  permet  I’utilisation 
d’une  application  graphique  TDSD  ecrite  en  GSQL 
(Graphical  Sequential  Query  Language)  qui  extrait 
et  manipule  directement  les  informations  de  la 
base.  Cette  approche  facilite  grandement  la  mise 
en  oeuvre  de  tris  et  filtres  appliques  aux  donnees. 

Les  mesures  de  performances  peuvent  etre 
visualisees  graphiquement  par  des  histogrammes 
tels  que  ceux  reproduits  a  la  Figure  5.  Ces 
resultats  represented  les  performances  obtenues 
par  une  poursuite  sur  le  meme  scenario  aerien, 
avec  des  environnement  electromagnetiques 
differents. 

Un  indice  de  satisfaction  peut  etre  attribue  a 
chaque  mesure  de  performance  (  de  1  a  10  par 
exemple)  de  telle  sorte  que  I’ensemble  des 
mesures  peut  etre  ensuite  ramene  a  une  note 
globale  par  ponderation  des  indices  en  fonctions 
des  criteres  de  pertinence  des  performances  vis  a 
vis  de  chaque  operateur.  Cette  methode  permet 
notamment  de  comparer  rapidement  plusieurs 
techniques  lorsque  I’analyse  detaillee  de  chaque 
mesure  peut  s’averer  extremement  fastidieuse. 


Figure  5:  Histogrammes  de  mesures 
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5^  UTILISATIONS  RECENTES 
Le  projet  RAPACE  a  ete  mis  en  oeuvre  pour  I’etude 
des  concepts  modernes  de  fusion  de  donnees  et 
d’identification.  Le  processus  de  selection  des 
soumissionnaires  pour  le  projet  devaluation  et  de 
demonstration  de  I’lDCP  (Identification  Data 
Combining  Process)  et  pour  I’appel  d’offre  de 
IACCS  LOCI,  ainsi  que  les  etudes  d’integration 
multicapteur  pour  la  renovation  de  la  capacity 
d’alerte  aeroportee  de  I’OTAN  sont  des  illustrations 
recentes  des  utilisations  de  ce  projet. 

6i_  DISCUSSION  SUR  LES  PERFORMANCES 

La  question  de  I’integration  de  capteurs  multiples 
dans  un  meme  systeme  de  fusion  de  donnees  est 
intimement  liee  avec  la  notion  de  performance.  La 
fusion  de  donnees  multicapteur  apporte  en  general 
une  solution  la  ou  un  systeme  monoradar  ne 
repond  plus  aux  exigences  operationnelles,  ou  du 
moins,  lorsqu’il  n’y  repond  plus  sous  certaines 
conditions.  Un  autre  facteur  d’integration  est  de 
permettre  une  degradation  progressive  en  cas  de 
panne  ou  de  dysfonctionnement  d’un  element  du 
reseau,  ou  encore  une  meilleure  resistance  aux 
contre-mesures.  On  attend  done  en  general  d’un 
systeme  multicapteur  de  meilleures  performances 
pour  assurer  les  memes  fonctions.  Cependant,  il 
est  clair  que  I’integration  multicapteur  autorise 
egalement  de  nouvelles  fonctionnalites  dont  le  gain 
est  plus  difficile  a  mesurer. 

La  fusion  multicapteur  illustre  egalement  la  notion 
de  “systeme”.  En  fait,  a  une  periode  ou  les  couts 
de  developement  logiciel  sont  en  baisse,  alors 
qu’inversement  les  recherches  et  developpements 
sur  des  capteurs  nouveaux  coutent  de  plus  en  plus 
cher  (I’amortissement  des  etudes  est  critique 
lorsque  le  nombre  de  capteur  produits  reste  limite), 
il  est  tentant  de  pallier  aux  problemes  de  detection 
par  des  techniques  informatique.  La  mise  a  profit 
de  la  probability  de  detection  cumulee  multicapteur, 
la  diminution  de  la  resolution  obtenue  par  le 
repartissement  geographique  des  capteurs,  ou 
encore  I’accroissement  le  la  cadence 
d’echantillonage  sont  autant  de  problemes  de 
detection  resolus  en  partie  par  la  fusion  de 
donnees.  Le  probleme  reste  a  traduire  ces  notions 
en  terme  de  performances  identifiables  et 
mesurables. 

Dans  une  elaboration  traditionnelle,  la  poursuite 
multicapteur  est  decrite  comme  un  processus 
purement  passif  a  boucle  ouverte,  traitant 
successivement  les  mesures  effectuees  sur 
I’environnement.  En  fait,  dans  un  contexte  elargi,  il 
convient  d’envisager  de  fermer  la  boucle,  e'est  a 
dire  d’agir  sur  les  capteurs  pour  optimiser  leur 
perception  de  I’environnement.  Le  controle 
intelligent  des  capteurs  a  partir  des  resultats  de 
poursuite  devrait  permettre  la  encore  d’ameliorer  la 


performance  globale  du  systeme  en  concentrant 
I’energie  et  en  commandant  des  formes  d’onde 
particulieres  sur  les  cibles  prioritaires  par  exemple, 
et  en  ayant  un  systeme  tres  adaptatif  a  son 
environnement. 

Des  travaux  sont  done  necessaires  pour  prototyper 
un  systeme  reactif  et  mettre  en  oeuvre  un 
environnement  de  simulation  qui  autorise  les 
traitements  de  poursuite  a  agir  sur  les  modeles  de 
capteurs.  Dans  un  tel  systeme,  I’analyse  de  la 
performance  aura  une  importance  encore  accrue 
puisque  cette  performance  sera  a  la  fois  un  but  et 
un  moyen  d'optimisation  du  systeme. 
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PAPER  No.  13 

DISCUSSOR’S  NAME:  M.  Bald 

COMMENT/QUESTION: 

Have  you  considered  any  civilian  standards  with  a  view  to  using  them  in  military  service  too?  As 
an  example,  the  ASTERIX  format  can  be  mentioned. 


AUTHOR/PRESENTER’S  REPLY: 

The  ASTERIX  format  has  been  foreseen  and  is  already  in  use  for  60th  ATC  and  military  radars  as 
far  as  surveillance  is  concerned.  There  is  still  some  discussion  on  how  to  incorporate  specific 

military  requirements  such  as  tactical  data  links,  sensor  control  orders,  etc .  Airborne  sensor 

systems  such  as  Joint  STARS  or  AWACS  do  not  have  equivalent  civilian  systems  to  share  data 
formats  with. 

DISCUSSOR’S  NAME:  M.  Bald 


COMMENT/QUESTION: 

What  is  the  relationship  between  the  performance  measurements  and  the  overall  system 
performance? 

AUTHOR/PRESENTERS  REPLY: 

Means  of  performance  can  be  either  technical  or  operational,  and  can  apply  either  one  sub¬ 
function  function,  or  on  the  whole  system,  according  to  the  level  of  detail  required. 


DISCUSSOR’S  NAME:  R.  Hogendoorn 

COMMENT/QUESTION: 

5  to  10%  of  the  measurements  are  not  taken  into  account  in  the  performance  measurements.  Are 
they  nevertheless  considered  as,  e.g.  a  measure  of  robustness? 


AUTHOR/PRESENTER’S  REPLY: 

The  discarded  fraction  corresponds  to  targets  performing  steep  manoeuvres  that  do  not  last  very 
long,  so  they  are  not  of  great  interest  for  an  operator.  Furthermore,  this  fraction  consists  of  false 
or  random  tracks  that  have  bad  MOPs  and,  thus,  disturb  the  MOP  figures.  However,  these  tracks 
are  looked  at  to  verify  that  they  belong  to  the  above  two  classes. 
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PAPER  No.  13 

DISCUSSOR’S  NAME:  J.L.  Maillart 

COMMENT/QUESTION: 

1 .  J’ imagine  que  la  modelisation  des  performances  dans  la  methode  analytique  necessite  de 
connaitre  exactement  le  detail  des  algorithmes  reels  ou  de  la  simulation:  dans  le  systeme 
developpe.  Est-ce  que  ce  sont  les  memes  personnes  qui  mettent  au  point  a  la  fois  les  algorithmes 
et  leur  modele  de  performances,  ou  bien  des  equipes  differentes? 

2.  Avez-vous  pu  comparer  les  resultats  obtenus,  pour  les  differentes  couches  de  traitement, 
entre  la  methode  de  simulation  et  la  methode  analytique? 

(1.  I  imagine  that  performance  modelling  in  the  analytical  method  requires  precise 
knowledge  of  the  real  algorithms  or  of  the  simulation:  in  the  developed  system.  Do  the  same 
people  design  the  algorithms  and  their  performance  models,  or  are  there  two  different  teams? 

2.  Have  you  been  able  to  compare  the  results  obtained,  for  different  layers  of  treatment,  by 

the  simulation  method  and  the  analytical  method?) 


AUTHOR/PRESENTER’S  REPLY: 

1 .  Ce  sont  les  memes  personnes  qui  pilotent  la  methode  analytique  sachant  que  les  niveaux 
de  detail  reclament  souvent  une  expertise  du  domaine  conceme  qui  est,  en  general,  fourni  par  des 
experts. 


2.  Les  travaux  confemant  l’approche  analytique  ne  sont  pas  suffisament  avances  pour  avoir 
un  recul  suffisant  concemant  la  possibility  de  confronter  les  methodes  analytiques  en 
confrontation  avec  les  resultats  de  simulation  ou  ses  donnees  reelles. 


(1.  The  same  people  work  on  both  methods,  although  the  detail  often  requires  special 

expertise,  which  is  provided  by  experts. 

2.  Work  on  the  analytical  approach  is  not  advanced  enough  to  give  any  perspective  on  the 
way  in  which  analytical  methods  could  be  compared  with  the  results  of  simulation  or  its  real 
data.) 


Reconnaissance  de  scenes  dynamiques  a  partir  de  donnees  issues  de 
capteurs :  le  projet  Perception 


Catherine  Tessier 
Onera  -  Cert 

2  avenue  Edouard-Belin  BP  4025 
31055  Toulouse  Cedex 
France 

20  aout  1997 


Resume 


L’objectif  du  projet  Perception  est  d’etudier  et  deve- 
lopper  des  methodes  permettant,  a  partir  de  l’observa- 
tion  de  l’environnement  par  differents  capteurs,  d’ela- 
borer  et  de  mettre  a  jour  une  representation  du  monde 
reel  propre  a  faciliter  une  prise  de  decision.  Le  projet 
contribue  ainsi  a  repondre  de  maniere  generique  aux 
besoins  operationnels  de  veille,  surveillance,  renseigne- 
ment,  reconnaissance,  que  ce  soit  dans  le  cadre  de  sys¬ 
temes  autonomes  ou  de  systemes  d’aide  a  la  decision, 
dont  le  fonctionnement  global  s’inscrit  dans  le  cadre 
d’une  boucle  fermee  perception  -  decision  -  action. 

La  fonction  de  perception  necessite  une  chaine  de  trai- 
tements  qui  s’etend  des  capteurs  jusqu’a  une  repre¬ 
sentation  semantiquement  riche  de  la  situation.  Cette 
chaine  est  rebouclee  pour  permettre  d’effectuer  une 
gestion  des  ressources  de  perception,  et  elle  fait  appel  a 
des  techniques  tant  numeriques  que  symboliques.  Elle 
est  organisee  autour  de  quatre  grands  modules:  les 
Traitements  Numeriques  ont  pour  objectif  de  delivrer 
des  objets  reconnus  dans  l’environnement  observe,  a 
partir  des  signaux  fournis  par  les  capteurs  ;  les  Traite¬ 
ments  Symboliques  ont  pour  but  d’elaborer,  a  partir 
des  objets  delivres  par  les  Traitements  Numeriques, 
la  ou  les  situations  en  cours  dans  l’environnement,  en 
vue  de  les  presenter  a  un  niveau  decisionnel ;  le  Modele 
d’Environnement  construit  et  met  a  jour  tous  les  ele¬ 
ments  pertinents  pour  la  representation  de  l’environ- 
nement  pergu ;  La  Gestion  du  systeme  de  Perception 
repond  aux  requetes  de  complements  d ’informations 
emises  par  les  Traitements  Symboliques  et  planifie  la 
prise  de  nouvelles  informations. 


1  Introduction 

L’objectif  du  projet  Perception  1[1]  est  d’etudier  et  de- 
velopper  des  methodes  permettant,  a  partir  de  l’ob- 
servation  de  l’environnement  par  differents  capteurs, 
d’elaborer  et  de  mettre  a  jour  une  representation  du 
monde  reel  propre  a  faciliter  une  prise  de  decision.  Le 
projet  contribue  ainsi  a  repondre  de  maniere  generique 
aux  besoins  operationnels  de  veille,  surveillance,  ren- 
seignement,  reconnaissance,  que  ce  soit  dans  le  cadre 
de  systemes  autonomes  ou  de  systemes  d’aide  a  la  de¬ 
cision,  dont  le  fonctionnement  s’inscrit  dans  le  cadre 
d’une  boucle  fermee  perception  -  decision  -  action. 


Fig.  1  -  la  boucle  perception  -  decision  -  action. 


La  fonction  de  perception  elabore  et  met  a  jour  une  re¬ 
presentation  semantiquement  riche  des  donnees  issues 
de  differents  capteurs  (cameras  noir  et  blanc  et  cou- 
leur,  cameras  infrarouges,  radar,  eventuellement  cap¬ 
teurs  “humains” )  disposes  dans  un  environnement  dy- 
namique.  Plus  precisement,  il  s’agit  de  fournir  des  ren- 
seignements  sur  les  actions  des  objets  en  mouvement, 
et  eventuellement  sur  leurs  intentions,  en  s’appuyant 
sur  des  modeles  spatio-temporels  de  haut  niveau  [2]. 

1.  consulter 

http:  /  /  www.cert.fr/francais/dera/dera/PERCEPT  ION  /per  ception.html 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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La  chaine  de  traitements  mise  en  oeuvre  s’etend  des 
capteurs  jusqu’a  une  representation  symbolique  de  la 
situation,  le  rebouclagepermettant  d’effectuer  une  ges- 
tion  des  ressources  de  perception.  Cette  chaine  est  or- 
ganisee  autour  de  quatre  grands  modules : 

•  les  Traitements  Numeriques  ont  pour  objectif  de  de- 
tecter  les  objets  en  mouvement  dans  l’environnement, 
a  partir  des  signaux  fournis  par  les  capteurs,  et  de  les 
reconnaitre. 

•  les  Traitements  Symboliques  ont  pour  but  d’elabo- 
rer,  a  partir  des  objets  delivres  par  les  Traitements 
Numeriques,  la  ou  les  situations  en  cours  dans  l’envi- 
ronnement,  en  vue  dc  les  presenter  a  un  niveau  deci- 
sionnel. 

•  le  Modele  d’Environnement  construit  et  met  a  jour 
tous  les  elements  pertinents  pour  la  representation  de 
l’environnement  perqu,  en  fonction  des  resultats  four¬ 
nis  par  les  Traitements  Numeriques  et  Symboliques. 

•  la  Gestion  du  systeme  de  Perception  (rebouclage) 
repond  aux  requetes  de  complements  d ’informations 
emises  par  les  Traitements  Symboliques,  en  utilisant 
le  Modele  d’Environnement  et  les  Traitements  Nume¬ 
riques.  II  planifie  la  prise  de  nouvelles  informations. 


bi-bande  3-5  pm  et  8-12  pm  en  champ  large,  deux  ca¬ 
meras  couleur  en  mode  stereoscopique  et  champ  large, 
une  camera  CCD  noir  et  blanc  en  champ  restreint.  Les 
scenarios  ont  ete  joues  de  jour  et  de  nuit. 

Les  scenarios  sont  de  trois  types : 

-  “feu”  :  ronde  de  surveillance  d’un  vehicule  de  pom¬ 
piers,  declenchement  d’un  feu  par  un  pyromane, 
extinction  du  feu  par  les  pompiers ; 


Fig.  3  -  une  scene  du  scenario  “feu” 


-  “circulation”  :  interaction  de  vehicules  et  de  pie- 
tons  donnant  lieu  a  differentes  occultations ; 


3 


Fig.  2  -  la  fonction  de  perception 


2  Les  scenarios  de  reference  :  les 
donnees  VlGlLE 

La  base  de  donnees  VlGlLE,  constitute  dans  le  cadre 
du  projet,  se  place  dans  le  cadre  general  de  la  sur¬ 
veillance  d’une  zone  semi-urbaine  par  un  systeme  au- 
tonome  mettant  en  ceuvre  plusieurs  capteurs  hetero¬ 
genes.  D’un  point  de  vue  pratique,  des  scenarios  ont 
ete  joues  sur  un  parking  entoure  de  batiments  et  de 
vegetation  et  les  donnees  ont  ete  recueillies  par  les  cap¬ 
teurs  suivants :  au  sol,  un  radar  94  GHz,  une  camera 
PtSi  3-5  pm  en  champ  moyen,  une  camera  analogique 
noir  et  blanc  en  champ  large  et  une  camera  CCD  cou¬ 
leur  en  champ  large ;  en  surplomb  de  la  scene  observee, 
une  camera  PtSi  3-5  pm  en  champ  etroit,  une  camera 


-  “agression  concertee”  :  encerclement  d’un  vehicule 
par  plusieurs  autres  vehicules  et  pietons,  mise  en 
evidence  d’un  comportement  collectif  agressif. 

L’objectif  de  ces  scenarios  est  d’illustrer  la  capacite 
de  la  chaine  de  traitements  a  prendre  en  compte  des 
points  durs  (saturation  de  certains  capteurs,  occulta¬ 
tions  intermittentes,  passageres  ou  definitives  d’objets, 
conflits  de  plans,  detection  d’un  comportement  global 
a  partir  de  comportements  individuels. . . )  tout  en  de- 
meurant  proche  d’une  certaine  realite  operationnelle. 
Meme  si  V IGILE  peut  apparaitre,  a  premiere  vue,  comme 
une  base  de  donnees  “jouet”,  un  examen  plus  appro- 
fondi  en  regard  de  scenarios  operationnels  montre  qu’une 
majorite  de  ses  caracteristiques  sont  pertinentes :  na¬ 
ture  des  objets,  types  de  scenarios,  distance  d’observa- 
tion,  dynamique  devolution,  masquage  des  objets  par 
des  elements  du  decor  ou  par  de  la  fumee.  Le  fait  que 
les  capteurs  soient  fixes  est  compense  par  la  simula¬ 
tion  de  prise  d’informations  partielles  (sur  des  fenetres 
d’images  ou  a  partir  de  capteurs  a  champ  restreint). 

3  Les  Traitements  Numeriques 
(TN) 

Les  Traitements  Numeriques  ont  pour  objectif  de  de- 
tecter  et  de  reconnaitre  les  objets  en  mouvement  dans 
1’environnement,  a  partir  des  signaux  fournis  par  les 
capteurs.  Dans  un  premier  temps,  le  capteur  radar 
n’est  pas  considere. 
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3.1  Calibration 

Les  signaux  provenant  de  plusieurs  cameras  hetero¬ 
genes  et  non  colocalisees,  la  premiere  etape,  pour  chaque 
scenario,  est  une  etape  de  calibration  geometrique  qui 
permet  d’associer  tout  point  d’une  image  2D  a  son  ho- 
mologue  dans  le  repere  3D  de  la  scene  observee,  dont 
on  dispose  d’un  plan  cote.  Cette  mise  en  correspon- 
dance  est  realisee  par  la  methode  de  Tsai  [3]. 

3.2  Detection  des  objets 

On  ne  s’interesse  qu’aux  objets  en  mouvement  car  eux 
seuls  apportent,  au  cours  du  temps,  une  information 
nouvelle  permettant  de  mettre  a  jour  la  connaissance 
deja  etablie  sur  la  scene. 

La  detection  est  effectuee  par  collaboration  entre  les 
algorithmes  de  detection  d’objets  en  mouvement  (al- 
gorithmes  “dynamiques”  ,  fondes  sur  le  calcul  du  gra¬ 
dient  spatio-temporel  de  plusieurs  images  successives) 
et  les  algorithmes  classiques  de  detection  (algorithmes 
“statiques”  fondes  sur  la  segmentation  de  l’image  cou- 
rante  en  regions  de  radiometrie  homogene).  L’interet 
de  cette  cooperation  est  d’allier  la  bonne  precision  ob- 
tenue  par  le  detecteur  de  contours  avec  la  bonne  ca- 
racterisation  obtenue  par  la  segmentation  en  regions, 
le  but  essentiel  etant  d’extraire  les  informations  nu- 
meriques  qui  vont  permettre  de  reconnaitre  un  objet 
parmi  un  ensemble  predefini  d’objets. 

Une  operation  de  detection,  qu’elle  soit  en  mode  in- 
frarouge  ou  couleur,  est  activee  par  une  requete  emise 
de  GP  vers  TN.  Cette  requete  contient  la  plage  tem- 
porelle  a  explorer,  la  zone  d’interet  a  scruter  dans  les 
images,  les  capteurs  a  selectionner. 

Detection  en  mode  infrarouge 

La  premiere  phase  du  traitement  consiste  a  detecter  un 
eventuel  changement  radiometrique  dans  l’image  cou- 
rante  par  rapport  a  l’image  precedente  ou  a  l’image 
suivante.  Ce  changement,  appele  phenomene,  est  uni- 
quement  de  nature  radiometrique.  Si  un  tel  change¬ 
ment  est  decele,  on  calcule  le  gradient  spatio-temporel 
de  trois  images  consecutives  It~i,It, h+ 1  ,  centrees  sur 
l’image  courante : 

GST{It-i,It,It+i)  =inf{\It-X  -Itl\It+i  -  h\)- 

Ce  gradient  est  ensuite  filtre  par  un  seuillage  par  hys¬ 
teresis  (le  seuil  a  une  valeur  haute  et  une  valeur  basse). 
Le  resultat  de  ce  seuillage  est  une  image  binaire  qui 
est  ensuite  filtree  par  une  fermeture  morphologique, 
qui  permet  de  connecter  des  regions  tres  voisines  mais 
non  connexes  initialement  a  cause  notamment  du  bruit 
dans  les  images.  Chaque  region  ou  composante  connexe 


correspond  a  un  phenomene  detecte. 

Dans  le  cas  des  images  VlGlLE,  le  contraste  thermique 
est  suffisant  entre  les  corps  froids  des  objets  du  decor 
et  les  corps  chauds  en  mouvement  pour  que  les  pheno¬ 
menes  detectes  correspondent  avec  une  quasi  certitude 
a  des  objets  en  mouvement. 

La  seconde  phase  consiste  a  affiner  le  resultat  prece¬ 
dent  en  appliquant  sur  l’image  binaire  une  methode 
de  croissance  de  regions.  Cette  etape  permet  de  deter¬ 
miner  avec  precision  la  forme  des  corps  chauds  et  en 
mouvement. 

Les  regions  ainsi  determinees  sont  etiquetees  et  carac- 
terisees  par  un  ensemble  de  parametres  radiometriques 
et  morphologiques,  en  vue  de  l’etape  de  reconnaissance 
des  objets. 

t  =  t+  l 


Segmentation 


Etiquetage  des  regions 
Caracterisation 


FlG.  4  -  detection  en  mode  infrarouge 

Detection  en  mode  couleur 

La  detection  de  phenomenes  est  realisee  selon  le  meme 
principe  que  precedemment,  avec  un  calcul  de  gradient 
spatio-temporel  prenant  en  compte  les  trois  compo- 
santes  rouge,  vert,  bleu : 

GST(It-i,It,It+i)  =  supc=R'VtB [inf(\ If_x  —  /tc | ,  | If+\  — 
TO 

En  revanche,  dans  le  cas  des  images  couleur,  on  ne  peut 
plus  affirmer  qu’un  phenomene  corresponde  toujours 
a  un  objet  en  mouvement.  La  methode  appliquee  est 
la  methode  de  detection  des  contours  d’objets  mobiles 
d’Orkitz  [4],  fondee  sur  le  calcul  d’un  gradient  spatio- 
temporel  a  partir  d’un  triplet  d’images  successives :  si 
At~i,  At  et  At+i  sont  les  amplitudes  respectives  du 
gradient  des  trois  images,  les  frontieres  en  mouvement 
sont  mises  en  evidence  par  le  gradient  Gt(x,  y) : 
Gt(x,y)  =  Max[At-i(x,y)-  At(x,y)]  At+1(x,y)} 


t-i 

i. 


t+i 


Detection  de 
phenomenes 


Phenomene  ? 
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- Max[At_i  ( x ,  y);  At+1  (x,  y )] 

Les  contours  sont  ensuite  fermes,  puis  1’etiquetage  des 
composantes  connexes  et  la  fusion  des  regions  adja- 
centes  sont  realises.  Comme  precedemment,  ces  re¬ 
gions  sont  etiquetees  et  caracterisees. 


t  =  t+  l 


Fig.  5  -  detection  en  mode  couleur 


peut  changer  de  type  (un  pieton  ne  se  transforme  pas 
en  vehicule)  -  mais  que  le  classifieur  peut  commettre 
des  erreurs  -,  et  que  des  attributs  de  type  couleur  ou 
forme  peuvent  varier  (en  fonction  de  l’eclairage  par 
exemple) .  Les  incertitudes  d’association  sont  modeli- 
sees  dans  le  cadre  de  la  theorie  des  possibilites. 
Exemple : 

pos 


i  pieton 

i 
i 

v(km/h) 
30 

pos 


vehicule 

v(km/h) 

Fig.  6  -  distribution  de  possibilites  sur  la  vitesse  des 
objets 


Classification  et  pistage  des  objets 

La  classification  est  realisee  essentiellement  sur  les  cri- 
teres  d’elongation  et  d’orientation  des  regions.  Elle 
s’appuie  sur  des  valeurs  de  reference,  apprises  sur  la 
base  d’images  Vigile  et  correspondant  aux  deux  classes 
d’objets  d’interet :  les  vehicules  et  les  pietons. 

De  maniere  generate,  les  pietons  sont  bien  reconnus  en 
mode  infrarouge;  en  ce  qui  concerne  les  vehicules,  la 
classification  est  meilleure  en  mode  couleur  (en  1 ’ab¬ 
sence  d’occultations),  quoique  plus  delicate  a  effectucr. 
En  sortie  de  classification,  une  fiche  Fn  est  emise,  avec 
les  renseignements  suivants  :  type  d’objet,  vraisemblance 
du  type,  date  tn  de  detection,  position  dans  l’environ- 
nement,  erreur  sur  la  position,  monochromacite,  cou¬ 
leur,  rotondite,  elongation,  orientation  principale,  am¬ 
plitude  et  orientation  de  la  vitesse. 

Le  pistage  a  pour  but  de  chainer  dans  le  temps  les 
objets  observes:  un  chainage  d’objets  est  une  piste. 
Chaque  arrivee  d’un  nouvel  objet  permet  soit  de  re- 
connaitre  la  continuation  d’une  piste  existante  et  done 
de  la  preciser,  soit  d’initialiser  une  nouvelle  piste. 

Un  objet  est  associe  a  une  piste  existante  si  ses  attri¬ 
buts  (les  differents  champs  de  la  fiche  Fn)  sont  compa¬ 
tibles  avec  ceux  de  la  piste.  Dans  le  cas  particulier  des 
images  VlGlLE,  on  tient  compte  du  fait  qu’un  objet  ne 


Etant  donne  un  objet  o  et  une  piste  p,  le  degre  de  pos¬ 
sibility  de  l’association  de  o  a  p  est  le  minimum  des  de- 
gres  de  possibility  d’association  compte  sur  l’ensemble 
des  attributs.  On  retient  les  associations  objet  -  piste 
dont  la  possibility  est  superieure  a  un  seuil  donne. 

4  Le  Modele  de  l’Environnement 
(ME) 

Le  Modele  de  l’Environnement  du  systeme  de  per¬ 
ception  est  constitue  de  l’ensemble  des  connaissances 
evolutives  relatives  a  1’environnement  observe  dont  la 
mise  a  jour  incombe  au  systeme.  C’est  egalement  une 
banque  de  donnees  dotee  d’outils  d ’interrogation  et  de 
mise  a  jour  au  service  des  autres  modules  du  systeme. 

Le  modele  geometrique  est  la  partie  du  modele  d’en- 
vironnement  qui  decrit  les  attributs  geometriques  de 
position  et  de  forme  des  objets  observes.  Dans  le  cas 
des  scenarios  Vigile,  il  decrit  les  elements  du  de¬ 
cor  (ces  connaissances  peuvent  etre  imprecises  et  done 
etre  mises  a  jour  au  fur  et  a  mesure  que  les  capteurs 
acquierent  de  l’information) ,  ainsi  que  les  vehicules 
et  pietons.  Chaque  objet  est  constitue  des  elements 
suivants:  un  identificateur  (identique  a  celui  donne 
par  TN  pour  les  objets  identifies  par  ce  module),  un 
type,  des  attributs  geometriques  (points  tridimension- 
nels,  aretes,  faces,  position  dans  un  repere  donne)  et 
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non  geometriques  (couleur,  temperature),  et  un  mo- 
dele  predictif  qui  permet  de  fournir  une  description  de 
l’objet  en  l’absence  d’informations  de  mise  a  jour.  Un 
historique  de  revolution  des  objets  et  de  leurs  attri- 
buts  au  cours  du  temps  est  conserve. 

Le  modele  permet  en  particulier  de  calculer  les  vues  at- 
tendues,  c’est-a-dire  l’estimation  de  ce  qu’un  capteur 
donne  pergoit  a  un  instant  donne. 


Fig.  7  -  le  modele  geometrique  superpose  a  une  image 


En  tant  que  base  de  donnees,  le  modele  d’environne- 
ment  peut  effectuer : 

-  des  operations  de  mise  d  jour  (modification  des 
attributs  des  objets  existants,  creation  de  nou- 
veaux  objets)  en  fonction  des  fiches  delivrees  par 
TN; 

-  des  operations  de  d’interrogation  en  fonction  des 
requetes  emanant  du  module  de  Gestion  de  la  Per¬ 
ception.  L’interrogation  peut  porter  sur  tous  les 
attributs  d’un  objet,  un  attribut  particulier  ou  un 
attribut  particulier  date. 

5  Les  Traitements  Symboliques 
(TS) 

A  partir  des  fiches  Fn  issues  de  TN,  les  Traitements 
Symboliques  [5],  [6],  [7]  elaborent  la  ou  les  situations  en 
cours  dans  l’environnement  observe,  c’est-a-dire  emettent 
des  hypotheses  sur  ce  que  font  les  objets.  Les  modeles 
de  reference  utilises  sont  des  activites  prototypes,  en¬ 
sembles  de  proprietes  attendues  pour  les  objets,  de- 
crites  par  des  cubes  logiques  (conjonction  de  formules 
logiques  atomiques  [8]),  et  des  plans  prototypes  repre- 
sentes  par  des  reseaux  de  Petri  interpretes  [9]  dont  les 
places  sont  associees  aux  activites ;  soit  V  l’ensemble 
des  plans  prototypes. 

Une  fonction  de  traduction  numerique/symbolique  trans¬ 
forme  d’abord  les  attributs  des  objets  des  fiches  Fn  en 


type(ol,  v6hicule) 
vitesse(ol,  v){v  £  0} 


activite 


O 


type(o2,  pi6ton) 
type(ol,  v6hicule) 

~1  proche-de(o2,  ol) 

deplacement-pieton 

depuis-vehicule 


'  marquage  atteignable 


condition  de 
terminaison  du  plan 


disparu(o2) 

Fig.  8  -  le  plan  prototype  “arrivee-en-vehicule” 


conjonction  de  proprietes  logiques. 

La  situation  courante  Sn  a  l’instant  tn  est  un  ensemble 
de  P-situations  definies  comme  des  reseaux 

de  Petri  marques  Pi  de  P ;  un  plan  prototype  Pi  peut 
etre  marque  par  le  marquage  m,  a  l’instant  tn  si  les 
proprietes  de  certains  objets  de  Fn  correspondent  a 
l’interpretation  de  certaines  places  de  Pj. 

Etant  donne  la  fiche  Fn+ 1  et  la  situation  courante  Sn , 
l’elaboration  de  la  situation  courante  Sn+i  a  l’instant 
tn+ 1  consiste  a  effectuer  une  mise  en  correspondance 
des  proprietes  des  objets  contenus  dans  Fn+ 1  avec 
les  marquages  atteignables  mj  +  k  des  P-situations 
(-Pi,m,',n)  (qui  correspondent  a  la  continuation  de  P- 
situations  existantes  -  (Pi:mi+k,n- k))  ;  ou,  si  certains 
objets  n’ont  pas  pu  etre  mis  en  correspondance  de 
cette  fagon,  a  creer  de  nouvelles  P-situations  ( Pj,mj  ,n+i ) 
Un  objet  donne  pouvant  etre  associe  a  plusieurs  P- 
situations,  Sn+i  est  l’ensemble  des  differentes  hypo¬ 
theses  de  plans  susceptibles  d’etre  en  cours  dans  l’en¬ 
vironnement  a  l’instant  tn+\ . 
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Fig.  9  -  le  module  de  Traitements  Symboliques 


Afin  d’affiner  l’elaboration  des  situations,  des  requetes 
d’informations  complementaires  a  celles  qui  figurent 
dans  les  fiches  sont  emises  de  TS  vers  GP. 

6  La  Gestion  de  la  Perception 
(GP) 

Le  module  de  Gestion  de  la  Perception  a  un  double 
role : 


-  il  regoit  les  requetes  de  TS,  interroge  ME  et  redi- 
rige  les  reponses  vers  TS ;  il  est  en  fait  l’interlo- 
cuteur  unique  des  autres  modules. 

-  il  elabore  les  strategies  de  prise  d’informations 
pour  TN,  selon  les  besoins  de  TS  et  en  fonction 
de  la  dynamique  propre  de  l’environnement. 

En  ce  qui  concerne  le  second  point,  il  s’agit  de  gene- 
rer  des  itineraires  de  prise  d’information  selon  deux 
modes :  un  mode  de  reponse  a  des  requetes  de  TS 
pour  lesquelles  ME  n’avait  pas  1’information  ou  une 
information  trop  ancienne;  un  mode  de  surveillance, 
dans  lequel  GP  cherche  a  mettre  a  jour  ME  en  tenant 
compte  des  evolutions  possibles  de  1’environnement  et 
de  P  erosion  des  croyances  sur  cet  environnement  (plus 
le  temps  passe,  plus  les  parties  de  l’environnement  qui 
ne  sont  pas  observees  sont  susceptibles  de  changer). 
Les  itineraires  dependent  done  de  l’etat  courant  de 
l’environnement :  une  distinction  explicite  est  faite  entre 
un  degre  de  confiance  representant  une  evaluation  de 
la  pertinence  de  l’estimation  courante  de  l’etat  de  l’en¬ 
vironnement  d’une  part,  et  l’imprecision  de  cette  es¬ 
timation  d’etat,  qualifiee  par  une  mesure  de  probabi¬ 
lity,  d’autre  part.  L’evolution  des  croyances  au  cours 
du  temps  est  alors  representee  formellement  selon  des 
principes  de  persistance  maximale  et  d’erosion  du  de¬ 
gre  de  confiance.  Les  regies  d’erosion  permettent  alors 
de  definir  des  strategies  de  perception. 

Les  zones  d’interet  possibles  qui  sont  identifies  au  ni¬ 
veau  du  modele  de  l’environnement  (calcul  des  vues 


attendues)  permettent  a  GP  de  designer  des  fenetres 
d’interet  dans  le  champ  de  vision  des  capteurs  [10] 


Definition  de  la  prochaine  fenetre 
d’observation  et  de  traitement 


Fig.  10  -  gestion  de  la  prise  d’information 


7  Traitement  d’un  scenario 

Supposons  pour  simplifier  que  l’ensemble  V  des  plans 
prototypes  ne  contienne  que  trois  elements:  arrivee- 
en-vehicule  (voir  Figure  8),  depart- en-vehicule  et  depla- 
cement-pieton  (voir  Figure  12). 

Le  modele  d’environnement  est  suppose  connaitre  le 
decor  et  les  objets  immobiles  (vehicules  stationnes  par 
exemple)  avant  le  demarrage  effectif  du  processus  de 
surveillance. 
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•  A  la  date  t\ ,  TN  fournit  la  fiche  F\ ,  qui  contient  un 
nouvel  objet  PI  tel  que  type(Pl,  pieton),  vitesse(Pl, 
4km/h). 


Fig.  11  -  imagel,  origine  de  la  fiche  Fi 


Aucune  P-situation  n’a  encore  ete  creee ;  les  plans  can- 
didats,  etant  donne  cette  seule  information  (pieton  en 
deplacement)  sont  les  trois  plans  de  V.  Pour  obtenir 
des  informations  complementaires,  TS  envoie  alors  des 
requetes  vers  GP  pour  demander  s’il  existe  un  vehicule 
dans  la  direction  de  marche  du  pieton  ou  proche  du 
pieton  (plan  depart-en-vehicule )  ou  s’il  existe  un  vehi¬ 
cule  dont  s’eloigne  le  pieton  (plan  arrivee-en-vehicule) . 
ME,  sollicite  a  son  tour  par  GP,  renvoie  les  reponses 
type(Vl,  vehicule),  vitesse(Vl,  0),  se-rapproche-de(Pl, 
VI)  e t  type(V2,  vehicule),  vitesse(V2,  0),  se-rapproche- 
de(Pl,  V2 )  a  la  premiere  requete  et  “echec”  aux  deux 
autres.  Les  P-situations  constituant  la  situation  Si 
peuvent  alors  etre  construites:  arrivee-en-vehicule  est 
rejete,  tandis  que  deplacement-pieton  et  depart-en-vehi¬ 
cule  sont  marques  respectivement  en  A1  (activite  “de¬ 
placement-pieton”)  et  A2A1  (activites  “deplaceinent- 
pieton- vers- vehicule”  et  “vehicule-stationne”). 


Dcpart_en_vehicule 


At 


0 


type(0_2,  pi6ton) 
vitesse(0_2,  v){v  ^  0} 


deplacement 

pieton 


disparu(0_2) 


Fig.  12  -  les  plans  prototypes  depart-en-vehicule  et 
deplacement-pieton 


•  A  la  date  t2,  avec  t2  =  t j  +  4s,  TN  fournit  la  fiche 
F2,  qui  contient  toujours  l’objet  PI  (qui  a  ete  piste) 
et  qui  est  maintenant  tel  que  proche-de(Pl,  VI). 

La  situation  S2  est  done  la  suivante :  l’activite  “deplace¬ 
ment-pieton”  du  plan  deplacement-pieton  est  toujours 
verifiee;  mais  deux  marquages  du  depart-en-vehicule 
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Fig.  13  -  image2,  origine  de  la  fiche  F2 


sont  maintenant  possibles :  A2A1  (activites  “deplace- 
ment-pieton-vers- vehicule”  et  “vehicule-stationne”)  et 
A3A1  (activites  “pieton-monte-dans-vehicule”  et  “vehi¬ 
cule-stationne”). 

•  A  la  date  t3,  avec  t3  =  t2  +  6s,  TN  fournit  une  fiche 
F3  vide  (pas  d’objet  en  mouvement  observable). 


Fig.  14  -  images,  origine  de  la  fiche  F3 

TS  emet  alors  cinq  hypotheses  de  P-situations  consti- 
tuant  la  situation  S3  :  en  ce  qui  concerne  le  plan  depla- 
cement-pieton,  l’activite  “deplacement-pieton”  est  soit 
terminee  (la  condition  de  terminaison  du  plan  est  ve- 
rifiee) ,  soit  occultee ;  en  ce  qui  concerne  le  plan  depart- 
en-vehicule,  trois  marquage  sont  possibles :  A2A1,  avec 
1’activite  “deplacement-pieton-vers-vehicule”  occultee, 
A3A1, 

avec  l’activite  “pieton-monte-dans-vehicule”  occultee, 
et  A4,  correspondant  a  l’activite  partiellement  inob¬ 
servable  “vehicule-stationne-occupe” . 

•  La  derniere  fiche  FA ,  delivree  a  la  date  f4 ,  avec  t4  = 
t3  +  14s,  contient  le  vehicule  VI  (piste  par  TN)  tel  que 
vitesse(Vl,  lOkm/h)  et  un  nouvel  objet  en  mouvement 
V2  identifie  comme  un  vehicule  tel  que  type(V2,  vehi¬ 
cule),  vitesse(V2,  30km/h). 

Le  vehicule  VI  permet  de  poursuivre  le  plan  depart-en- 
vehicule,  avec  le  marquage  A5  (activite  “demarrage- 
vehicule”). 

En  ce  qui  concerne  V2,  un  nouveau  plan  doit  etre 


Fig.  15  -  image 4,  origine  de  la  fiche  F4 


considere:  arrivee-en-vehicule  est  candidat,  instancie 
en  P-situation  par  le  marquage  Al  (activite  “deplace- 
ment-vehicule-depuis-entree-parking” ) .  La  situation  S4 
met  done  en  jeu  une  conjonction  de  deux  P-situations. 

8  Conclusions 

Les  differents  modules  de  la  chaine  de  traitements  sont 
implantes  sur  differents  calculateurs,  dans  des  langages 
differents,  et  dialoguent  entre  eux  par  echange  de  mes¬ 
sages  Ascii  par  protocole  sockets.  GP,  qui  a  un  role 
central,  est  le  seul  module  a  devoir  traduire  les  mes¬ 
sages  regus  et  envoyes. 

Les  travaux  en  cours  portent  sur  differents  approfon- 
dissements  a  apporter : 

-  amelioration  de  la  classification  des  objets  (base 
d’apprentissage,  fusion  des  classifieurs  infrarouge 
et  couleur) ; 

-  introduction  de  connaissances  de  TS  dans  le  pis- 
tage; 

-  modele  d’environnement  “cooperatif”  (definition 
d’explications  raisonnables  a  des  objets  non  iden¬ 
tifies  ou  a  l’absence  ou  la  disparition  d’objets ;  eta- 
blissement  d’informations  complementaires  perti- 
nentes) ; 

-  robustesse  de  la  reconnaissance  de  situation :  de¬ 
finition  d’un  cadre  algebrique  pour  le  traitement 
de  l’incertitude  symbolique  [7] ; 

-  pilotage  effectif  des  TN  a  partir  de  GP. 
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SUMMARY 

This  paper  describes  a  system  for  surveillance 
and  target  recognition  for  autonomous  stand-off 
weapons  based  on  the  interpretation  of  multisensor 
data  in  combining  the  individual  sensor  channels  to 
optimize  the  system  performance  by  intelligent  fu¬ 
sion  techniques. 

To  meet  this  demand,  we  propose  a  model-based 
method  for  the  automatic  recognition  of  high  value 
fixed  or  relocatable  surface  targets  like  bridges,  air¬ 
fields,  industrial  installations  and  command  posts. 
A  production  net  is  used  to  represent  the  know¬ 
ledge  about  target  structures.  The  analysis  is  car¬ 
ried  out  by  a  knowledge  based  classification  system. 

Starting  with  primitive  objects  extracted  sepa¬ 
rately  from  each  individual  sensor  channel  (low- 
level-processing),  more  complex  parts  are  composed 
step  by  step  until  the  target  object  is  recognized 
(high-level-processing).  The  radar  preprocessing 
comprises  a  thresholding  prescreener  and  a  subse¬ 
quent  feature  based  discrimination  stage,  the  fea¬ 
tures  making  use  of  polarimetric  and  geometric 
properties,  and  of  scattering  center  statistics.  The 
IR-preprocessing  is  more  related  to  geometric  and 
structural  features  like  line  elements,  corners  and 
circles.  To  enhance  the  detection  probability,  in  the 
low  level  stage  a  high  false  alarm  rate  is  accepted 
as  regards  the  primitive  objects.  The  reduction  of 
the  false  alarm  rate  is  done  in  the  high  level  stage 
by  the  spatial  fusion  of  2D-  or  3D-structures  and 
by  using  additional  context  information  of  the  tar¬ 
get  environment. 

The  purpose  of  this  paper  is  to  address  the  state 
of  the  algorithms  for  automatic  recognition  of  high 
value  surface  targets.  For  this,  sensor  data  have 
been  interpreted  which  were  recorded  with  an 
imaging  infrared  sensor  and  a  coherent,  polarimet¬ 
ric,  high  range  resolution  radar.  To  enable  the  spa¬ 
tial  fusion  process,  the  raw  sensor  data  were  regis¬ 
tered  together  with  inertial  data  of  the  aircraft. 

The  efficiency  of  the  analysis  system  is  demon¬ 
strated  by  an  example  involving  the  detection  of 
command  posts. 

The  results  show  the  suitability  of  the  method  for 
future  autonomous  stand-off  weapon  systems  like 


drones  and  missiles  with  imaging  sensors. 

1.  INTRODUCTION 

Future  military  systems  for  surveillance  and  target 
recognition  will  increasingly  be  based  on  the  inter¬ 
pretation  of  multisensor  data.  The  amount  and 
complexity  of  the  data  will  exceed  the  human  abil¬ 
ity  to  interpret  the  data  creating  a  demand  for  au¬ 
tomation  of  all  or  part  of  the  fusion  process.  In 
the  case  of  stand-off  weapons  (missiles  and  drones 
[1,2]),  human  interaction  is  removed  entirely  from 
the  decision  loop  and  the  processing  has  to  be  fully 
automated. 

One  of  the  main  research  activities  of  FGAN  is  the 
development  and  evaluation  of  methods  for  auto¬ 
matic  target  acquisiton  in  multisensor  data. 

For  the  detection  of  specific  high  value  fixed  or 
relocatable  surface  targets  we  propose  a  model- 
based  structural  analysis  method  using  geometric 
and  generic  target  information  and  additional  con¬ 
text.  In  the  low-level  processing  stage  a  high  false 
alarm  rate  of  primitive  objects  making  up  a  target 
is  accepted  to  enhance  the  detection  probability. 
The  reduction  of  the  false  alarm  rate  is  done  in  the 
high-level  processing  stage  by  the  spatial  fusion  of 
more  complex  2D-  or  3D-structures  and  the  inter¬ 
pretation  of  context  information  like  map  and  situ¬ 
ation  information.  The  process  of  building  up  more 
complex  structures  from  less  complex  structures  is 
implemented  by  a  production  system.  In  this  way, 
the  system  performance  is  based  on  a  proper  scene 
analysis  and  it  is  more  independent  of  an  optimal 
selection  of  algorithms  and  the  associated  parame¬ 
ters  than  conventional  system  design. 

The  model-based  approach  has  been  applied  suc- 
cesfully  to  the  detection  of  different  high  value 
fixed  target  classes  like  bridges  and  airfields 
[3, 4, 5, 6].  This  paper  describes  the  efficiency  of  the 
analysis  system  for  the  detection  of  command  posts 
which  belong  to  the  class  of  high  value  relocatable 
targets. 

Due  to  the  non-availability  of  the  final  sensor  sys¬ 
tem,  which  is  still  under  development  (section  2.1) 
sensor  data  have  been  analysed  which  were  taken 
by  an  imaging  infrared  sensor  (8-12 pm)  and  a  co- 
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herent,  polarimetric,  high  range  resolution  radar 
(94GHz)  in  a  side  looking  synthetic  aperture  ap¬ 
proach. 

This  work  is  closely  connected  to  current  exper¬ 
imental  studies  of  future  weapon  systems  as  di¬ 
rected  by  the  German  MoD. 

2.  BACKGROUND 

2.1  Dualmode  Seeker  System  ABG 

The  development  of  an  autonomous  missile  sys¬ 
tem  for  the  detection  and  engagement  of  command 
posts  (ABG)  is  identified  as  an  important  task 
by  the  military  defense  authorities.  The  studies 
are  carried  out  by  German  companies.  BGT  and 
SIEMENS.  The  FGAN  research  institutes  are  in¬ 
volved  in  an  advisory  capacity  concerning  the  def¬ 
inition  of  sensor  parameters  and  the  evaluation  of 
data  fusion  algorithms  for  target  detection. 

The  activities  started  with  the  definition  of  the  ex¬ 
pected  scenario  together  with  the  inclusion  of  the 
availability  of  aerial  reconnaissance  information  for 
target  description  and  mission  planning.  Based 
on  these  requirements,  the  seekerhead  system  de¬ 
sign  and  system  parameters  were  determined  for 
the  construction  of  an  experimental  seekerhead  sys¬ 
tem.  In  parallel,  signal  processing  methods  for  sen¬ 
sor  data  fusion  have  been  developed.  During  the 
next  phase  of  the  project,  captive  flight  tests  will 
be  realized  to  demonstrate  the  suitability  of  the 
dualmode  sensor  and  to  evaluate  the  overall  system 
performance. 

The  results  of  these  studies  lead  to  a  concept  for  a 
seekerhead  system,  which  provides  a  dualmode  sen¬ 
sor  consisting  of  an  imaging  infrared  sensor  with 
a  focal  plane  array  detector  and  a  real  aperture 
stepped  frequency  pulse  radar.  Both  sensors  are 
integrated  within  a  common  aperture  and  the  sys¬ 
tem  is  provided  with  an  antenna,  scanning  in  az¬ 
imuth  and  elevation  direction.  The  system  is  also 
equipped  with  an  Antiradiation  (AR)-component 
which  is  not  further  considered  in  this  paper. 

The  experimental  seekerhead  system  is  still  under 
construction  and  will  be  evaluated  by  captive  flight 
tests  starting  in  1998. 

2.2  Dualmode  Data  Sources 

Due  to  the  lack  of  real  ABG  sensor  data  at  present, 
the  development  of  target  detection  algorithms  is 
implemented  with  simulated  data.  Several  measure¬ 
ment  campaigns  have  taken  place  with  different  ex¬ 
isting  sensor  systems  to  gather  data  of  the  quality 
similar  to  that  of  the  later  ABG  sensor  hardware. 

Radar  data  was  collected  by  using  the  experimen¬ 
tal  radar  system  MEMPHIS  (—  Millimeter  wave 
Experimental  Multifrequency  Polarimetric  High 
resolution  Imaging  Sensor)  of  FGAN-FHP  [7,8]. 
This  is  a  coherent,  polarimetric,  high  range  reso¬ 
lution  radar,  operating  at  a  nominal  frequency  of 


94GHz,  which  has  been  installed  on  board  an  air¬ 
craft  in  a  side-looking  SAR  configuration.  In  [9]  the 
measurement  system  is  described  and  some  evalua¬ 
tion  procedures  with  emphasis  on  SAR  preprocess¬ 
ing  are  discussed.  The  radar  is  equipped  with  a 
dual  channel  receiver  to  register  simultaneously  the 
parallel  and  orthogonal  polarization  component  of 
the  backscattered  signal.  The  radar  data  together 
with  flight  parameters  (pitch,  roll,  heading,  time 
code)  were  gathered  by  means  of  fast  acquisition 
electronics  and  written  onto  a  magnetic  tape.  For 
image  analysis  the  resulting  radar  images  were  in¬ 
terpreted  as  2-dimensional  images. 

Fig.  1  shows  an  example  of  the  examined  radar 
images  (scene  A).  The  scene  contains  a  command 
post  with  various  deployed  elements,  whereas  the 
most  obvious  part  of  the  target  is  located  next  to 
a  big  building  in  the  upper  part  of  the  SAR  image. 
It  consists  of  a  cluster  of  vehicles  (jeeps,  light  and 
heavy  trucks,  power  generators)  covered  by  cam¬ 
ouflage  nets  along  the  lower  side  and  around  the 
lower  corners  of  the  building.  The  surroundings 
of  the  target  area  consist  of  grass-land  with  many 
scattered  trees  and  bushes.  Moreover,  there  are 
numerous  artificial  man-made  objects  in  the  scene, 
like  sheds,  mounds,  various  small  buildings  and  un¬ 
camouflaged  vehicles. 


Fig.  1:  SAR  Image 

The  IR  data  were  gathered  with  the  experimental 
dualmode  sensor  system  MSS  [10].  This  system  is 
equipped  with  an  imaging  infrared  sensor  (8-12 fim) 
and  a  FMICW  monopulse  radar  (94GHz)  and  is 
provided  with  a  scanning  antenna.  Because  of  the 
poor  lateral  resolution  of  the  MSS  radar  data  (ver¬ 
sus  the  expected  data  of  the  ABG  system),  the 
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mmW  channel  is  not  considered.  For  the  data  col¬ 
lection  the  sensor  frontend  was  fixed  forward  to  a 
transport  helicopter  under  a  variable  depression  an¬ 
gle.  Sensor  and  flight  parameters  were  recorded 
during  the  measurements.  The  scene  is  scanned  in 
a  pushbroom  way  producing  a  sequence  of  overlap¬ 
ping  images.  Because  of  the  narrow  field-of-view  of 
the  sensor,  a  single  image  contains  only  a  small  sec¬ 
tion  of  the  scene. 

To  get  a  complete  coverage  of  the  target  area  and 
to  simulate  the  future  ABG  sensor  characteristics  it 
is  necessary  to  register  the  individual  IR-frames  in 
a  ground  map.  Fig.  2  shows  the  result  of  the  regis¬ 
tration  process. 


Fig.  2:  IR  Ground  Map 


2.2  Data  Harmonization  for  ABG  Simulation 

IR-  and  SAR-data  were  recorded  using  different 
carriers  with  different  flight  parameters.  As  con¬ 
sequence,  only  a  portion  of  the  scene  is  covered 
by  both  sensors.  To  enable  the  data  fusion  pro¬ 
cess,  both  data  sets  have  to  be  transformed  into  a 
common  reference  coordinate  system.  This  is  done 
by  an  affine  transformation  of  the  radar  data  into 
the  infrared  image  representation  neglecting  special 
3D-effects  of  the  different  image  generation  process 
(Fig.  3).  These  data  are  stored  in  the  database  of 
the  simulation  test  bed  (section  3.1)  together  with 
the  data  of  temporal  and  geometrical  harmoniza¬ 
tion  and  could  be  addressed  for  analytical  purposes 
as  originating  from  the  real  ABG  sensor. 

Therefore  the  harmonization  process  is  not  basic  to 
the  sensor  fusion  process,  because  the  detection  al¬ 
gorithms,  described  in  section  3,  are  designed  to 
take  into  consideration  that  in  the  real-time  system 
the  infrared  information  is  gathered  frame  by  frame 
and  the  radar  data  line  by  line,  so  that  processing 
runs  synchronized  with  data  acquisition. 


Fig.  3:  Harmonization  of  IR  and  SAR  Data 

3.  MODEL-BASED  SENSOR  FUSION 

3.1  The  FIM  BPI-Sensor  Fusion  Test  Bed 

For  the  task  of  target  detection  in  multispectral 
data  the  Blackboard-based  Production  system  for 
Image  understanding  (BPI)  is  used  [11,12,13]. 

The  BPI-System  is  a  framework  for  model-based 
structure  analysis  of  complex  scenes  and  consists  of 
three  basic  components:  a  global  database  (black¬ 
board),  a  control  unit  and  a  set  of  processing  mod¬ 
ules  (agents).  Fig.  4  shows  the  blackboard  architec¬ 
ture  adapted  for  target  classification  in  dualmode 
sensor  data. 


Fig.  4:  System  for  Target  Recognition 
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The  analysis  starts  with  a  low-level-processing  or 
preprocessing  of  the  sensor  raw  data  by  extracting 
proper  primitive  objects  independently  in  each  sen¬ 
sor  channel.  Objects  with  higher  complexity  (par¬ 
tial  objects)  of  the  searched  target  will  be  created 
by  applying  production  rules  of  the  medium-level¬ 
processing.  The  inference  process  of  the  model- 
based  sensor  fusion  analysis  will  combine  these  in¬ 
termediate  results  to  classify  the  final  target  by 
high-level-processing  which  uses  production  rules  of 
higher  complexity. 

All  objects,  intermediate  or  final,  represent  certain 
structures  of  the  target  and  are  stored  with  suit¬ 
able  attributes  in  the  database  which  functions  as 
an  associative  memory.  The  productions  (” rules”) 
are  implemented  by  processing  modules  which  com¬ 
municate  exclusively  via  the  blackboard.  The  con¬ 
trol  unit  monitors  the  system  activities.  The  anal¬ 
ysis  is  data-driven  and  is  designed  for  running  in 
parallel.  Details  concerning  the  architecture  and 
the  dataflow  of  the  BPI-System  are  described  in 

[14]- 

In  addition  to  the  modules  for  the  dualmode  tar¬ 
get  recognition,  modules  for  background  analysis 
(e.g.  texture  analysis  [15])  and  the  automatic  in¬ 
terpretation  of  auxiliary  information  (situation  and 
map  analysis)  are  integrated  into  the  BPI-System. 
It  is  expected  that  this  additional  context  informa¬ 
tion  (e.g.  neighbourhood  relations  to  objects  of  the 
background)  will  enhance  the  detection  probability. 

For  a  dualmode  seeker  system  with  the  option  of 
target  engagement,  modules  for  homing  (e.g.  track¬ 
ing  algorithms)  have  to  be  incorporated. 

In  general,  the  fusion  of  multisensor  data  is  based 
on  a  spatial  confirmation  of  geometrical  target  fea¬ 
tures  in  a  world  reference  system.  For  this  spatial 
fusion  it  is  necessary  to  know  the  process  of  im¬ 
age  generation,  which  enables  the  transformation  of 
sensor  data  into  the  reference  system  (registration). 
In  particular,  the  sensor  parameters  must  be  avail¬ 
able  which  consist  of  the  extrinsic  (position,  ori¬ 
entation)  and  intrinsic  (focal  length,  range  resolu¬ 
tion...)  sensor  parameters. 

3.2  Modelling 

The  general  interactions  of  productions  and  the 
stepwise  transfer  of  objects  into  objects  of  higher 
complexity  can  be  displayed  by  a  production  net 
[16].  It  represents  the  target  model  or  the  know¬ 
ledge  about  the  target. 

For  our  basic  research  we  define  as  an  example  a 
generic  command  post  consisting  of 

•  cluster  of  vehicles 

•  power  generators 

•  containers 

•  antennas 


which  often  are  installed  close  to  buildings  (in  ur¬ 
ban  areas)  or  which  are  partly  obscured  e.g.  in 
clearings  (in  wooded  areas). 

Consequently,  basic  signatures  that  can  be  ex¬ 
pected  -  with  respect  to  the  sensor  resolution  at 
the  proper  distance  -  are  contours  (lines),  ar¬ 
eas  (edges),  hotspots  (cues)  and  similar  structural 
parts  of  the  target  components  depending  on  the 
signature  in  the  individual  sensor  channel.  For  each 
part  of  these  structures  (object  parts),  a  proper  set 
of  attributes  is  processed. 

Maps,  aerial  photos  (if  available)  or  comparable 
information  like  military  studies  can  be  used  as 
knowledge  sources  for  the  geometric  arrangement 
of  these  primitive  objects  making  up  the  target. 
By  this  the  model  generation  process  needs  surveil¬ 
lance  in  advance. 

The  general  procedure  of  modelling  is  illustrated 
by  the  production  net  of  fig.  6  using  the  command 
post  of  test  scene  A  (fig.  5)  as  an  example.  The 
productions  determine  how  a  given  set  of  objects  is 
transferred  into  a  set  of  more  complex  objects. 

In  our  example,  the  command  post  is  composed  of 
a  cluster  of  vehicles  and  power  generators  situated 
near  a  building.  The  building  is  represented  by  its 
roof  which  appears  in  aerial  images  as  a  parallelo¬ 
gram. 


Fig.  5:  Reference  Image 

Starting  with  the  primitive  objects  Line,  the  ob¬ 
jects  Angle  are  built  by  objects  Line  enclosing  an 
angle  a.  If  two  objects  Angle  form  a  structure  like 
an  open  parallelogram  they  are  combined  to  an  ob¬ 
ject  U_Structure.  An  object  Parallelogram 
can  be  built  if  objects  U_Structure  and  Line  are 
compatible.  Objects  Parallelogram  are  actually 
used  to  describe  buildings,  because  in  aerial  im¬ 
ages  containing  houses  only  their  roofs  can  be  rec¬ 
ognized  which  often  appear  as  parallelograms. 

For  the  construction  of  vehicle  clusters,  primitive 
objects  Cues  are  combined  to  Pairs  if  their  dis- 
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Fig.  6:  Model  for  Object  COMMAND  Post 


tance  corresponds  to  the  model  parameter.  Two 
objects  Pairs  are  grouped  to  an  object  Cluster  if 
they  have  a  common  endpoint.  In  a  similar  way  ob¬ 
jects  Cluster  are  produced  by  objects  Pairs  and 
objects  Cluster  or  objects  CLUSTER  and  objects 
Cluster. 

Finally  the  target  object  Command  Post  is  gener¬ 
ated  by  an  object  Cluster  situated  in  the  neigh¬ 
borhood  of  an  object  Parallelogram. 

With  this  model  and  the  use  of  known  flight  and 
sensor  parameters  the  geometric  structure  of  the 
command  post  and  its  parametric  description  can 
be  computed  at  the  actual  detection  time. 


3.3  Model-Based  Multi  Sensor  Fusion 
3.3.1  IR-Preprocessing 

The  basic  elements  for  the  structure  analysis  are 
the  primitive  objects  Line  and  Cue  (fig.  6).  They 
are  created  in  the  preprocessing  stage  by  standard 
iconic  image  processing  methods.  In  this  way  sym¬ 
bolic  descriptions  of  the  images  are  generated. 

The  method  for  generating  the  objects  LINE  is 
based  on  a  contour  approximation.  These  lines  are 
extended  to  longer  lines. 

The  choice  of  the  cue  detection  algorithm  is  guided 
by  the  spatial  resolution  of  the  object.  For  the  de¬ 
tection  of  point  targets  a  hotspot  operator  is  used 
based  on  a  nonlinear  filtering. 

As  we  have  a  scanning  sensor  providing  an  im¬ 
age  sequence,  the  IR-preprocessing  algorithms  are 
applied  to  each  individual  image.  For  the  con¬ 
struction  of  more  complex  objects,  lines  and  cues 
are  registered  in  a  world  reference  system  and  are 
stored  in  the  database  as  primitive  objects  LINE 
and  Cue  respectively. 


3.3.2  SAR-Preprocessing 

Similar  to  the  IR-preprocessing  the  generation  of 
primitive  objects  in  the  radar  channel  is  carried  out 
by  a  radar  specific  method.  As  input  for  sensor  fu¬ 
sion,  the  object  CUE,  as  candidate  for  vehicle,  and 
the  object  LINE,  as  a  basic  element  of  man-made 
objects  are  considered;  they  are  extracted  from  the 
data  in  two  steps. 

The  first  step  consists  of  an  amplitude  based 
thresholding  prescreener  to  select  potential  target 
areas  and  to  reduce  the  computational  burden  in 
the  subsequent  processing  stages.  Two  prescreener 
types  are  defined  which  work  on  single  range  pro¬ 
files.  The  first  uses  the  total  power  contrast  to 
determine  areas  of  potential  target  (vehicle)  sites. 
The  second  type  of  prescreener  represents  a  shadow 
filter  and  detects  strongly  reflecting  structures  cast¬ 
ing  a  shadow  like  vehicles  and  edges  of  buildings. 

Based  upon  the  output  of  the  prescreener,  feature 
vectors  are  then  constructed  in  the  second  step  to 
discriminate  between  cues  and  natural  clutter  back¬ 
ground.  For  this  purpose,  several  polarimetric  and 
geometric  features  are  computed  over  small  areas 
centered  at  the  cell  under  test.  Only  those  cells 
that  had  passed  the  prescreener  were  subjected  to 
the  feature  test.  The  size  of  the  areas  is  adapted 
to  the  size  of  the  target  i.e.  the  discriminator  acts 
on  a  small  number  of  range  profiles.  This  takes  into 
account  the  fact,  that  in  the  real  ABG-sensor  sys¬ 
tem,  the  incoming  radar  backscatter  signals  form  a 
time  sequence.  That  means  that  each  detection  al¬ 
gorithm  has  to  start  its  work  while  the  radar  image 
is  still  being  constructed. 

For  the  generation  of  the  primitive  objects  Line 
the  same  procedure  is  applied  as  described  above 
for  the  extraction  of  lines  in  the  IR  channel.  The 
lines  are  extracted  in  a  moving  window  of  the  SAR 
map  corresponding  to  the  consecutive  accumulation 
of  range  profiles.  In  combination  with  the  results 
of  the  shadow  filter,  lines  with  high  reflecting  parts 
are  candidates  for  man-made  objects. 

The  final  result  is  an  image  that  contains  within 
each  pixel  the  decision  Cue,  if  it  is  a  candidate  for 
vehicle  and  a  symbolic  description  of  objects  Line 
for  all  candidates  of  buildings.  Details  concerning 
this  radar  specific  preprocessing  are  described  in 
[3,9,17]. 

Fig.  7  shows  the  results  which  are  stored  as  primi¬ 
tive  objects  to  be  processed  in  the  following  model- 
based  data  fusion. 

3.3.3  Application  to  Dualmode  Sensor  Data 

The  general  task  of  the  ABG  sensor  is  to  analyse 
a  certain  area  of  interest,  obtained  via  a  priori  in¬ 
formation,  and  to  identify  autonomously  command 
posts  within  that  area  and  define  an  adequate  aim- 
point  on  that  target.  In  doing  this,  special  con¬ 
straints  defined  by  the  different  phases  of  mission 
(midcourse  guidance,  acquisition,  recognition,  en¬ 
gagement)  have  to  be  taken  into  consideration. 
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Cues  (black  squares) 
Fig.  7:  SAR 


Lines  (black  lines) 
Low-Level  Results 


During  midcourse  guidance,  the  missile  is  steered 
to  the  target  area  by  INS  with  image  based  naviga¬ 
tion  update.  This  part  of  the  mission  is  not  further 
addressed  in  this  paper. 

By  approaching  the  predefined  target  area,  the  ac¬ 
quisition  phase  is  initiated,  which  is  characterized 
by  scanning  the  scene  under  a  low  depression  an¬ 
gle.  By  that,  a  large  search  area  is  covered  by 
the  sensor  with  a  medium  spatial  resolution  of  ob¬ 
jects  in  the  IR  channel.  At  the  beginning  of  the 
acquisition  phase,  the  application  of  signal  process¬ 
ing  methods  should  result  in  cueing  and  selecting 
areas  of  interest ,  which  are  potential  sites  of  com¬ 
mand  posts.  This  action  can  be  accompanied  by 
autonomous  flight  path  correction  to  optimize  the 
data  gathering  for  target  detection. 

After  the  acquisition  phase,  the  areas  of  interest 
are  interpreted  in  more  detail  during  the  target 
recognition  phase.  Therefore,  the  spatial  resolution 
of  objects  in  the  IR  channel  is  enhanced  by  increas¬ 
ing  the  depression  angle  continously.  At  the  same 
time,  the  size  of  the  scanning  area  is  reduced.  At 
the  end  of  this  phase,  locations  of  candidates  for  a 
command  post  are  determined  to  initialize  the  en¬ 
gagement  phase.  The  overall  detection  process  can 
be  interpreted  as  an  active  vision  procedure. 

Starting  with  cueing  structures  of  poor  complexity, 
areas  of  interest  are  determined,  which  are  anal¬ 
ysed  in  the  next  processing  stage  in  more  detail. 
This  results  in  cueing  objects  with  higher  complex¬ 
ity  and  the  definition  of  more  precise  areas  of  inter¬ 


est  (e.g.  with  smaller  size)  to  be  analysed  in  the 
following  stage.  The  analysis  ends  when  the  desired 
degree  of  complexity,  precision  of  localisation  and 
confidence  of  target  identification  is  reached. 

In  every  stage  of  the  analysis  process  the  complex¬ 
ity  of  modelling  and  the  selection  of  the  processing 
method  have  to  be  adapted  to  the  degree  of  target 
resolution.  Moreover,  in  the  real  system,  the  detec¬ 
tion  process  can  be  supported  by  an  active  control 
of  the  sensor  parameters  (scan  angle  and  rate,  de¬ 
pression  angle  etc.)  and  the  flight  path  of  the  mis¬ 
sile. 

Fig.  8  shows  the  result  of  the  model-based  struc¬ 
ture  analysis  generated  during  a  simulated  acqui¬ 
sition  phase  of  test  scene  A.  The  objects  Clus¬ 
ter  (white  polygons)  are  built  up  by  objects  Cue 
(white  squares),  which  were  extracted  in  the  IR- 
and  radar  channel.  Candidates  for  buildings  (black) 
are  represented  by  objects  U_Structure  and  ob¬ 
jects  Parallelogram  produced  by  primitive  ob¬ 
jects  Line,  which  were  extracted  in  the  individual 
sensor  channels.  All  objects  are  assessed  according 
to  a  spatial  confirmation  of  the  associated  infrared 
and  radar  structures. 

The  area  of  interest  (white  rectangle)  is  determined 
by  the  area  surrounding  the  best  assessed  candi¬ 
date  for  the  object  COMMAND  POST  (black)  pro¬ 
duced  by  the  objects  Cluster  (black)  situated 
near  candidates  for  buildings  (objects  Parallelo¬ 
gram). 


Fig.  8:  Area  Of  Interest 

In  the  next  step,  the  target  recognition  phase  is 
simulated  by  a  detailed  analysis  of  individual  IR- 
frames  belonging  to  the  area  of  interest.  Fig.  9 
shows  a  sequence  of  6  consecutive  IR-frames  (area 
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of  interest)  covering  parts  of  the  potential  target 
while  scanning.  The  primitive  objects  CUE  (white 
squares)  and  Line  (black  lines),  detected  in  the 
individual  frames,  deduce  the  object  Parallel¬ 
ogram  (white)  and  the  object  CLUSTER  (white 
polygon)  and  together  form  the  object  Command 
Post,  which  is  the  highest  level  of  complexity  of 
the  target  model  in  the  implemented  sensor  fusion 
process. 

The  results  are  visualized  in  an  infrared  panoramic 
representation. 


Frame  ni-riQ  IR-Panorama  Representation 


Fig.  9:  Detail  Analysis 


4.  CONCLUSIONS 

The  FGAN  research  institutes  FIM  and  FHP  are 
involved  in  the  development  of  dualmode  sensor  fu¬ 
sion  technologies.  A  main  task  consists  of  advis¬ 
ing  the  German  MoD  on  the  evaluation  of  system 
performance,  sensor  characteristics  and  algorithm 
quality.  The  design  of  systems,  sensor  hardware 


and  the  integration  of  algorithms  for  realtime  ap¬ 
plication  are  tasks  of  the  companies  and  are  not 
part  of  our  activities. 

For  the  evaluation  of  multisensor  data  fusion  algo¬ 
rithms,  the  BPI  test  bed  has  been  developed.  The 
examinations  carried  out  with  data  obtained  from 
a  multisensor  system  have  shown  that  knowledge- 
based  structure  analysis  for  autonomous  detection, 
classification  and  identification  of  special  targets  is 
appropriate  for  seekerhead  and  drone  applications. 

The  objective  of  the  ongoing  research  is  the  ex¬ 
tension  of  the  system  to  a  three-dimensional  fusion 
and  interpretation  of  multisensor  data.  In  addi¬ 
tion,  procedures  for  texture  analysis  will  be  incor¬ 
porated  in  the  system  for  the  interpretation  of  con¬ 
text  information.  It  is  assumed,  that  these  exten¬ 
sions  will  improve  detection  and  target  localization 
performance. 
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SUMMARY 


The  present  work  is  addressed  to  perform  an 
estimation  in  an  accurate  and  robust  way  of  the 
trajectory'  of  a  land  vehicle  by  using  a  Differential 
Global  Positioning  System  (DGPS)  and  an  Inertial 
System  (INS). 

The  use  of  a  Kalman  Filter  (KF)  approach  for 
integration,  data-fusion  and  estimation  tasks  has 
been  proved  as  able  to  providing  precise  and  robust 
evaluation  of  cinematic  variables  (linear  position  and 
velocity)  even  in  the  case  of  long  missions  or  under 
critical  conditions  of  temporary  incompleteness  or 
unreliability  of  part  of  the  acquired  data. 

From  the  state  of  the  art,  it  can  be  seen  that  the  DGPS 
is  a  very  precise  sensor  providing  3D  geographic 
position,  but  presents  low  output  rate  and  temporary' 
signal  loss  or  accuracy  degradation,  while  the  INS 
provides  continuous  outputs  of  rotation  angles  and 
linear  acceleration  with  high  output  rate  but  the 
inertial  units  have  burdensome  intrinsic  errors  which 
bring  about  a  degradation  of  precision  increasing  with 
time. 

Practically,  the  integration  of  DGPS  and  INS  is 
forecast  to  provide  continuous  estimates  over  time, 
corrupted  by  small  and  almost  unchanging  errors. 

The  system  has  been  tested  over  an  extensive  set  of 
real  data  providing  good  results  both  in  precision  and 
in  robustness. 

1,  INTRODUCTION 

In  the  last  years,  an  increasing  interest  for  the 
development  of  systems  able  to  compute  the  precise 
position  of  an  object  in  the  ground  has  been  shown. 

In  this  work,  a  system  for  the  determination  of  the 
position  of  a  vehicle  based  on  the  integration  of  two 
different  sensors,  DGPS  and  INS,  is  proposed. 

The  integration  and  estimation  approach  is  based  on 
the  use  of  the  Kalman  Filter. 


The  control  of  the  position  and  the  tracking  of  a 
vehicle  equipped  by  a  set  of  localization  sensors  and 
instruments  for  the  estimation  of  cinematic  variables 
are  a  very  difficult  problem. 

In  this  context  Kalman  Filter  (KF)  has  intensively- 
been  used  in  navigation  systems  since  1960.  Most 
recent  applications  concern  w'ith  the  use  of  KF  in  the 
satellite  navigation  systems,  such  as,  for  example,  the 
Global  Positioning  System  (GPS). 

One  of  the  most  interesting  problems  is  the 
integration  of  inertial  systems  (gyroscopes, 
accelerometers,  i.e.  Inertial  Navigation  Systems  - 
INS)  with  other  Navigation  Systems,  for  example 
GPS. 

In  fact,  inertial  systems  are  characterised  by  some 
drift  phenomena  that  cause  the  increasing  of  the 
errors  at  the  increasing  of  the  time  according  to  less 
or  more  complex  laws. 

Due  to  this  fact,  during  long  missions,  inertial 
systems  have  to  be  used  together  with  other 
navigation  systems  whose  aim  is  to  support  INS  in 
the  global  navigation  system. 

The  present  work  aims  at  estimating,  in  an  accurate 
and  robust  way,  the  trajectory'  of  a  land  vehicle  by 
using  the  Differential  Global  Positioning  System 
(DGPS)  and  the  Inertial  Navigation  System  (INS). 

In  the  most  of  cases,  the  estimation  of  a  vehicle 
position  achieved  by  KF  cannot  be  optimal  due  to  the 
fact  that  the  noise  corrupting  the  chosen  status’ 
variables  is  Non-Gaussian. 

In  this  paper,  a  new  dynamic  model  for  KF  equation 
has  been  introduced,  allowing  one  to  have  only 
Gaussian  variables  corrupting  the  status’  variables 
and  consequently  to  obtain  an  optimal  position 
estimation  of  the  vehicle. 

The  proposed  system  is  also  able  to  manage  the 
different  reliability'  of  the  DGPS  data,  thanks  to  the 
adaptation  of  KF  parameters  to  the  different 
situations. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 


16-2 


The  paper  is  organized  as  follows:  first  of  all  the 
characteristics  of  DGPS  and  INS  are  presented  and 
possible  integration  approaches  are  described. 

In  the  following,  the  architecture  of  the  proposed 
system  is  presented  and  most  important  concepts 
concerning  the  new  integration  approach  are  shown. 
Finally,  some  experimental  results  obtained  in  the 
position  estimation  of  the  vehicle  are  illustrated. 

2.  THE  DGPS  SIGNAL 

The  Global  Positioning  System  (GPS)  [1]  is  a 
navigation  system  based  on  satellite  communications 
use  that  allows  the  user  to  determine  its  position  in 
every  ground  site  which  is  not  occluded  by  eventual 
obstacles. 

GPS  offers,  with  every  atmospheric  condition  and  at 
every  instant  of  the  day,  an  instantaneous  and  global 
vehicle  positioning,  with  a  few  meters  error, 
depending  on  the  receivers'  cost. 

GPS  is  currently  employed  in  many  application 
fields:  for  example,  industrial  applications  concern 
with  surveillance,  maps  creation,  photogrammetry, 
public  security,  telecommunications. 

The  precision  error  of  the  GPS  may  be  reduced  to 
less  than  one  meter,  by  means  of  the  introduction  of 
differential  correction  techniques  [2], 

The  Differential  GPS  (DGPS)  is  an  improvement  of 
GPS  that  uses  errors  introduced  by  GPS  in  the 
localisation  of  known  points  in  order  to  correct  the 
errors  in  the  localisation  of  unknown  positions. 

DGPS  is  a  very  precise  sensor  that  gives  3D 
geographic  positions,  but  that  presents  a  low  output 
rate  (<  1  Hz)  and  either  temporary  signal  losses  or 
accuracy  degradation. 

The  main  disadvantage  of  DGPS  is  the  occlusion  of 
the  signal  by  buildings,  tunnels  or  trees:  in  such  cases 
the  number  of  visible  satellites,  on  which  the  DGPS 
signal  is  based,  decreases  and  the  positioning 
provided  by  the  DGPS  is  not  trustful. 

3.  THE  INS  SIGNAL 

Inertial  Navigation  System  (INS)  is  a  system  for  the 
estimation  of  a  vehicle  position,  which  is  based  on 
the  Newton  laws  of  the  classical  mechanics. 

The  vehicle  position  is  obtained  by  means  of  a  double 
integration  of  the  measured  vehicle  acceleration. 

Three  accelerometers  and  three  gyroscopes,  aligned 
with  respect  to  axes  mutually  orthogonal,  which 
constitute  an  inertial  reference  system,  measure  the 
linear  acceleration  and  the  angular  speed  of  the 
vehicle  [3J. 

The  gyroscopes  also  provide  the  necessary 
information  in  order  to  stabilise  the  platform  on 
which  sensors  have  been  mounted  with  respect  to  the 


chosen  inertial  reference  system.  The  described 
stabilisation  can  be  performed  both  physically,  within 
a  system  called  gimbal  [4],  both  analytically,  in  the 
so-called  strap-down  [5]  systems.  After  an  initial 
start-up,  the  INS  begins  to  produce  a  continuous 
output  of  the  position,  speed  and  direction  of  the 
vehicle,  independently  from  every  external  agent  and 
from  environmental  conditions. 

The  INS  provides  continuous  outputs  of  rotation 
angles  and  linear  accelerations  at  high  output  rates 
[10,  50]  Hz,  by  reducing  sampling  errors  and  by 
improving  the  precision. 

The  INS  advantage  lies  in  its  independence  from 
external  factors  and  in  the  fact  that  from  the  INS 
outputs  position,  speed  and  orientation  angles  are 
immediately  available.  Moreover,  in  the  short  time, 
INS  is  more  precise  than  GPS,  by  a  10  factor  [6], 

Main  disadvantages  of  INS  are  its  relevant  cost  and 
the  fact  that,  if  it  is  not  integrated  by  some  other 
positioning  methods,  INS  has  an  intrinsic  time- 
varying  error  that  causes  a  rapid  deterioration  in  the 
precision  of  the  position  estimation.  Such  error 
corresponds  to  around  2  Km  for  each  navigation 
hour. 

4.  WHY  INTEGRATION  OF  GPS  AND  INS  ? 

The  general  motion  of  a  vehicle  in  a  three- 
dimensional  space  can  be  described  by  means  of  six 
parameters:  three  accelerations  and  three  orientation 
angles. 

A  system  for  measuring  the  trajectory  of  a  vehicle  has 
to  evaluate  six  independent  measured  sizes,  from 
whose  it  has  to  derive  the  already  mentioned 
parameters. 

GPS  signal  integrated  by  INS  signal  can  be  used  in 
order  to  achieve  such  a  purpose. 

It  can  be  demonstrated  that  the  GPS  shows  a 
significative  superiority  with  respect  to  the  INS  in 
terms  of  accuracy  and  precision,  given  that  inertial 
sensors  are  affected,  in  the  long  term,  by  relevant 
errors. 

Moreover,  GPS  has  an  higher  cost  than  INS  and  it 
does  not  require  an  initialisation  procedure. 
Nevertheless,  also  GPS  has  certain  limits  and  INS  has 
some  useful  characteristics,  in  particular  the 
continuous  output  of  position,  speed  and  direction  of 
the  vehicle:  in  fact,  INS  has  outputs  independent  from 
external  agents,  such  as  the  satellites  visibility  or  the 
meteorological  conditions. 

Due  to  these  facts,  the  integration  of  GPS  and  INS,  is 
able  to  offer,  as  final  result,  a  system  which  allows  to 
maintain  advantages  of  both  navigation  techniques 
and  to  remove  many  disadvantages  which  could  be 
present  if  GPS  and  INS  were  singularly  considered 
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Hence  the  integration  of  DGPS  and  INS  data  is 
forecast  to  provide  continuous  estimates  over  time, 
corrupted  by  small  and  almost  unchanging  errors. 

5.  THE  KF-BASED  INTEGRATION 
APPROACHES 

The  main  objective  of  the  integration  between  DGPS 
and  inertial  data  is  to  mix  all  the  available 
information  provided  by  sensors  in  order  to  obtain  the 
estimation  of  the  position,  speed  and  direction  of  a 
vehicle:  this  will  allow  to  reconstruct  the  trajectory  of 
the  vehicle.  - 

The  sy  nchronisation  of  data  coming  from  DGPS  and 
the  ones  computed  by  the  inertial  navigation  system 
is  essential  in  order  to  correctly  compare  all  the 
information  concerning  the  moving  vehicle. 

In  general,  INS  can  be  used  in  order  to  interpolate  the 
position  estimated  by  DGPS  with  a  very  high 
frequency  and  in  order  to  replace  the  DGPS  signal 
when  it  is  obscured  or  in  the  case  of  bad  satellites 
configuration. 

Most  commonly  used  technique  for  DGPS  and  INS 
data  processing  and  fusion  is  Kalman  Filtering. 


However,  this  filtering  technique  is  not  the  solution  to 
all  problems  introduced  by  the  DGPS/INS 
Integration:  in  fact,  one  that  has  to  design  such  a 
system  must  take  into  account  some  specifications 
that  have  to  be  respected  in  order  to  obtain  a  sy  stem 
that  is  really  operating. 

These  requirements  mainly  are  computational 
efficiency,  precision  of  the  estimation,  robustness  to 
different  kinds  of  noise,  compatibility  with  different 
systems:  however,  the  recursive  characteristic  of  the 
Kalman  Filter  makes  it  efficient  for  real-time 
applications. 

Also  its  noise  robustness  is  a  characteristic  necessary 
for  real-time  navigation  systems. 

In  the  following  paragraphs,  the  proposed  approach 
based  on  the  use  of  Kalman  Filter  will  be  presented: 
the  peculiarities  of  the  proposed  approach  will  be 
highlighted  and  some  experimental  results  obtained 
in  the  estimation  of  the  position  of  a  land  vehicle. 

6.  SYSTEM  DESCRIPTION 

The  KF-based  approach  used  for  the  DGPS/INS 
integration  is  based  on  a  tight  integration  scheme  [7]: 
the  structure  of  the  system  is  shown  in  Figure  1. 


Gobal 

Positioning 

System 

Reoeiver 


Figure  1.  KF-based  architecture  for  DGPS/INS  tight  integration 
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which  can  be  obtained  through  the  averaging  in  a 
unique  value  five  successive  INS  samples. 


By  observing  Fig.  1  it  is  possible  to  see  that: 

•  a  unique  Kalman  Filter  is  used  to  process  signals 
coming  from  satellites  and  inertial  ones; 

•  GPS  based  positioning  is  directly  integrated  by 
the  INS  outputs: 

•  DGPS  and  INS  are  strictly  integrated  and  the 
performances  of  the  two  systems  become  strictly 
dependent  each  other. 


In  fact,  the  capability  of  DGPS  to  “follow”  the 
vehicle  when  the  vehicle  performs  sharp  manoeuvres 
will  depend  on  the  accuracy  of  the  INS  outputs.  On 
the  other  part,  the  INS  accuracy  will  depend  on  the 
frequency  and  on  the  quality'  of  the  updating 
produced  by  DGPS. 

In  such  a  way,  if  measures  coming  from  DGPS  are 
not  trustful,  they  can  be  refused  in  a  unique  time, 
without  compromising  tire  performances  in  the 
position  calculation,  thanks  to  INS  data. 

On  the  counterpart,  if  the  INS  signal  is  wrong  and  the 
DGPS  signal  is  absent  or  not  trustful,  the  system  is 
not  able  to  establish  the  exact  position  of  the  vehicle, 
unless  a  dedicated  Kalman  Filter  for  only  DGPS  data 
processing  is  inserted. 

Following  the  proposed  approach,  the  position  of  the 
vehicle  is  estimated  through  the  use  of  INS  data  to 
calculate  the  trajectory  of  reference  and  of  DGPS 
data,  if  available,  to  update  the  trajectory  itself. 

Before  describing  the  structure  of  the  used  Kalman 
Filters  based  architecture  and  of  the  real  DGPS/INS 
integration,  it  could  be  useful  to  expose  which  is  the 
main  strategy  of  the  DGPS/INS  integration  itself. 

At  a  first  approximation,  dynamic  variables  in  the 
status  vector  of  the  system  model  could  be  the  three 
error  variables  obtained  by  means  of  the  difference 
between  the  position,  the  speed  and  the  acceleration 
estimated  by  DGPS  and  the  ones  calculated  by  INS, 
that  is: 

•  J  =  (5r  5v  8a)  (1) 

Such  error  variables,  which  are  estimated  by  KF,  are 
then  used  to  update  the  INS  output  and  to  generate 
the  position  of  the  vehicle.  It  is  important  to  recall  the 
different  rate  of  DGPS  data  (1  Hz)  and  of  INS  data 
(10  Hz). 

Because  of  the  sensors  configuration,  system  will 
have  to  read  one  DGPS  datum  even'  ten  INS  read 
data.  The  integration  system  produces  as  output  an 
estimation  of  the  vehicle  position  at  a  rate 
corresponding  to  2  Hz. 

Due  to  this  fact,  it  will  be  necessary  to  insert  a  pre¬ 
processing  step  on  DGPS  data,  by  interpolating  two 
successive  samples  in  order  to  obtain  a  2  Hz  DGPS 
output  and  by  making  an  INS  data  compression. 


The  proposed  system,  as  previously  said,  uses  DGPS 
produced  data  as  a  reference  to  continuously  update 
the  inertial  navigation  system. 

In  such  a  way,  an  eventual  absence  of  DGPS  signal, 
due  to  satellites  problems  or  to  obstacles  placed 
between  users  and  satellites  themselves,  could  be 
compensated  through  the  use  of  a  INS  signal  which 
can  be  considered  as  trustful,  at  least  in  the  short 
periods. 

The  estimation  of  the  (dr  dv  da)  vector,  w  hich  is 
calculated  by  KF,  will  be  subtracted  from  the  current 
values  of  position,  speed  and  acceleration  calculated 
by  INS  and  it  allows  to  update  the  values  of  INS 
outputs 

Unfortunately,  the  updating  offered  by  DGPS  is 
sometimes  not  trustful  or  absent  at  all. 

The  KF-based  system  has  been  designed  in  order  to 
make  it  possible  to  also  adopt,  besides  to  the  basic 
fusion  strategies  (DGPS  present,  DGPS  absent),  other 
operating  possibilities,  in  order  to  take  into  account 
the  characteristics  of  the  DGPS  signal:  DGPS  quality 
can  be  characterized  by  a  reliability  degree  depending 
on  the  value  of  a  parameter  called  Position  Dilution 
of  Precision  (PDOP)  [7], 

Such  “Multi-Hypothesis”  approach  makes  the 
proposed  system  able  to  commute  betw  een  different 
operating  possibilities,  according  to  the  reliability  of 
DGPS  data  and  consequently  to  the  value  of  the 
PDOP  parameter. 

In  such  a  way,  PDOP  becomes  the  control  variable 
depending  on  which  to  apply  the  different  available 
integration  strategies. 

The  proposed  integration  system  is  also  able  to 
follow  the  vehicle  even  in  the  worst  case,  when 
DGPS  is  absent. 

In  this  case,  the  status  vector  constituted  by  the 
error/updating  variables  estimated  by  KF  is  obtained 
through  subtracting  current  the  cinematic  measures 
calculated  by  INS  from  the  ones  estimated  by  the 
system  at  the  previous  instant. 

In  the  following  paragraph,  the  proposed  data  fusion 
approach,  that  allows  to  find  the  optimal  estimation 
of  the  vehicle  position  through  the  integration  of 
DGPS  and  INS,  is  presented. 
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7.  THE  PROPOSED  DATA  FUSION 
APPROACH 

The  peculiarity  of  the  employment  of  the  Kalman 
Filter  in  the  proposed  approach  mainly  resides  in  the 
noise  modeling  approach. 

In  fact,  it  is  known  that  a  KF  gives  optimal  results 
only  if  noise  affecting  status  and  measures  vectors  is 
Gaussian. 

In  this  case,  such  assumption  is  not  valid,  due  to  the 
fact  that  tire  models  of  the  noise  acting  on  the 
accelerometers  and  on  the  gyroscopes  are  not  linear 
and  they  have  non-Gaussian  statistics. 

In  fact,  taking  into  account  only  the  system  dynamics 
and  the  final  estimation  targets,  the  variables  in  the 
status  vector  X  —  (5 r  5v  5a)  ,  are  three  error 
variables  carried  out  by  the  differences  between  the 
positions,  velocities  and  accelerations  provided  by  the 
DGPS  and  the  INS. 

Moreover,  one  of  the  main  critical  aspects  of  the 
DGPS/INS  integration  is  the  deviation  from 
Gaussianity  of  some  of  the  noise  components 
corrupting  the  status  variables'  true  values: 
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The  term  [5/''  Sv'  8ci' j  is  the  time  derivative 
of  the  differential  status  variables,  /  is  a  [3x3]  identity 
matrix. 

The  term  Aa.  that  is  one  introduced  by  inertial 
sensors,  can  be  identified  as  the  error  affecting  the 
INS  measured  speed  and  can  be  modelled  as  non- 
Gaussian  noise. 

In  this  case,  the  estimation  which  could  be  obtained 
from  a  system  defined  in  such  a  way  should  be  only 
sub-optimal,  due  to  the  fact  that  a  KF  is  able  to 
guarantee  the  optimal  estimation  only  when  additive 
Gaussian  noises  are  present  [8  ], 

In  order  to  obtain  an  estimation  which  could  be 
approximately  considered  as  optimal  it  is  necessary  to 
perform  some  modification  on  the  status  model:  such 
modifications  should  have  to  take  into  account  the 
non-Gaussian  noise  components  corrupting  the 
outputs  of  the  system. 

Therefore,  assuming  this  term  to  be  ideally  Gaussian 
might  degrade  the  system  performances.  [8] 

In  this  paper,  a  new  approach  that  allows  to  reach  the 
optimality  of  the  position  estimation  is  proposed:  in 
particular,  through  this  approach,  the  optimality  is 


achieved  through  the  linear  modelling  of  the  error 
present  on  sensors  through  the  techniques  described 
in  the  following  and  the  extension  of  the  dimension 
of  the  Kalman  status  vector. 

At  this  end,  the  dynamic  variables  which  characterise 
the  behaviour  of  the  error  present  in  the  sensors, 
conveniently  corrupted  by  white  noise  with  known 
statistics,  will  be  added  to  the  status  vector  itself. 

It  has  also  to  be  pointed  out  that  different  strategies  in 
order  to  perform  the  optimal  estimation  w  ill  be 
adopted  according  to  the  PDOP  value,  that  is 
according  to  the  reliability  of  DGPS  data. 

Finally,  the  estimated  error  will  be  then  subtracted 
from  the  INS  data,  in  order  to  update,  on  the  basis  of 
the  more  reliable  DGPS  data,  the  inertial  navigation 
system. 

In  order  to  achieve  optimal  estimates  under  these 
critical  conditions,  a  two-step  algorithm  is  proposed. 

The  first  step  was  an  accurate  error  analysis,  in  order 
to  establish  the  components  having  the  higher 
influence  in  the  cinematic  status  variables  and  to 
define  their  dynamic  model. 

Successively  it  has  been  possible  to  extend  the 
dimensions  of  the  status  vector  by  adding  the  noise 
components  and  by  linearly  modelling  their 
dynamics. 

The  dynamic  system  should  be  linear  and  corrupted 
by  Gaussian  noise:  then,  it  should  be  possible  to 
calculate,  through  the  Kalman  approach,  an  optimal 
estimation  of  the  status  vector. 

Let  us  briefly  expose  the  approaches  that  have  been 
followed  in  order  to  achieve  an  optimal  estimation  of 
the  status  vector. 

8.  THE  ERROR  MODELING  CHOSEN 
APPROACH 

The  first  step  to  be  covered  in  order  to  find  the 
optimal  estimation  of  the  vehicle  position  is  to  define 
an  appropriate  model  of  the  non-Gaussian  noise 
corrupting  the  status'  variables  of  the  system. 

The  main  components  of  this  noise  are  due  to 
accelerometer  and  gyroscope  errors  and  they  are 
called  drifts. 

The  errors  of  accelerometers  are  modelled  as  being 

composed  of  bias  (b  ),  random  ramp,  random  walk 
and  a  first-order  Markov  process  (in )  with  a  time 
constant  T  ,  being  X  a  vector  composed  by  three 
components  |ta.  Ty  Z",  J. 

The  chosen  model  of  the  drift  rate  error  9  satisfies 
the  following  equations: 
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where  Aa  is  the  acceleration  drift,  r  is  the  ramp 
slope,  S  is  the  sum  of  bias,  random  walk  and  ramp 
and  C  is  the  white  Gaussian  noise  [9], 

The  same  model  can  be  applied  to  the  gyroscope 
drifts. 


Actually,  in  the  definition  of  (3),  components 
modelled  by  the  first-order  Markov  process  have  not 
been  taken  into  account. 

Because  of  this  fact,  the  model  of  the  error  corrupting 
the  outputs  of  the  accelerometer  can  be  defined  in  the 
following  way. 


Let  Aa  be  the  error  corrupting  the  output  of  the 


accelerometer  (drift)  and  let  Aa '  its  derivative  with 

respect  the  time  (drift  rate)-,  drift  can  be  defined  as  a 
vector  composed  by  components  at  constant  error 


components 


(i) 


\  J 


and  by  components  given  by  the 


f- 


sum  of  bias,  random  walk  and  ramp 
following  w  ay : 


V 


S  ,  in  the 


(4) 


The  model  of  the  error  affecting  accelerometers  has 
then  be  modelled  as: 


where  B  is  defined  as: 


0 

0 


(5) 


(6) 


model  of  the  status  of  the  system,  by  increasing  the 
dimensions  of  the  status  vector  in  order  to  take  into 
account  the  behaviour  of  the  modelled  error 
components. 

In  such  a  way  the  status’  variables  vector  will  be 
only  corrupted  by  a  Gaussian  noise  and  the  optimal 
estimation  of  the  vehicle  position  will  be  possible. 


9.  EXTENDED  STATUS  VECTOR  FOR  NON- 
GAUSSIAN  NOISE  MODELLING  AND 
FILTERING 


The  status  vector  of  the  Kalman  Filter  is  augmented 
by  adding  to  it  the  status  variables  that  model  non- 
Gaussian  noise  components,  so  that  the  augmented 
final  status  vector  will  be 

x  -  (8r  8v  da  r  s  in)T 

Now'  the  system  is  assumed  to  be  corrupted  only  by 
white  Gaussian  noise,  so  we  can  achieve  the  optimal 
estimate. 

The  dynamic  variables  added  to  the  status  vector 
concern  only  with  the  accelerometer  drift  model. 

The  drift  rate  of  the  gyroscope  is  evaluated  by  using  a 
parallel  standing-alone  KF,  whose  resulting  estimate, 
V]/' ,  is  used  to  compute  the  rotation  rate  of  the 
vehicle  {(O ),  needed  by  the  integration  filter  to 
express  the  inertial  acceleration  at,  provided  by  the 
INS,  in  terms  of  a  non-inertial  acceleration  ani 
comparable  with  the  one  provided  by  the  DGPS 
(application  of  the  3D  motion  equations  for  a  rigid 
body)  [10], 


while  ^  is  a  white  Gaussian  noise. 


Once  the  dynamic  behaviour  of  the  Non-Gaussian 
noise  corrupting  the  outputs  of  the  accelerometers  has 
been  modelled,  it  is  then  possible  to  re-definc  the 
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The  resulting  status  model  is: 

8Ff  0/00 

8v'  00/0 

8a'  0  0  0  0 

r'  =  0  0  0  0 

s'  0  0  0  / 

rn’  0  0  0  0 


where  are  white  Gaussian  noise  components. 

The  observation  equation  is  }’  =  Hx  +  D  ,  y  being 
the  obsen  ation  vector,  H  the  measurement  matrix 
and  U  the  vector  of  Gaussian  noise: 

7  0  0  0  0  0 

‘  y=(dr  5v  5a) ;  77=  0  7  0  0  0  0 

0  0  7  0  0  0 

(8) 

The  estimated  position,  velocity  and  acceleration 
errors  computed  by  the  KF,  which  now  represent  an 
optimal  estimation  of  the  status  variables,  are  then 
subtracted  from  the  INS  current  position,  velocity  and 
acceleration,  thus  updating  the  values  of  the  INS 
outputs. 

In  such  a  way,  the  optimal  trajectory  of  the  vehicle 
can  be  obtained  through  the  integrated  use  of  the 
DGPS  and  INS  signals. 

Unfortunately,  this  integration  scheme  is  valuable 
only  when  the  updating  measures  provided  by  the 
DGPS  are  reliable. 

In  order  to  solve  this  problem  a  multi-hypothesis 
approach  for  the  integration  has  been  introduced  and 
in  particular  the  KF  estimates  have  been  made 
dependent  of  the  value  of  the  PDOP  parameter,  which 
is  the  one  that  gives  the  reliability  index  of  the  DGPS 
data. 

10.  THE  MULTI-HYPOTHESIS  CHOSEN 
APPROACH 

The  Multi-hypothesis  approach,  which  has  been  used 
in  the  proposed  approach  in  the  integration  between 
DGPS  and  INS  data,  is  a  particular  strategy  that 
allows  to  take  into  account  the  variable  goodness  of 
the  measure,  provided  by  the  DGPS. 


In  such  a  w'ay,  an  eventual  absence  of  the  DGPS 
signal,  due  to  satellites  problems  or  to  some  obstacles 
located  between  the  user  and  the  satellites,  could  be 
compensated  through  the  use  of  the  INS  signal,  that, 
in  the  low  or  medium  term,  can  be  considered  as 
trustable. 

The  estimation  of  the 

X  =  (dr  dv  da  F  s  m)  vector,  that 
is  calculated  by  the  KF,  will  be  subtracted  to  the 
current  values  (e  g  position,  speed  and  acceleration) 
given  by  INS  and  the  values  of  the  outputs  of  the 
inertial  navigation  system  will  be  updated. 

The  KF-based  system  has  been  designed  in  such  a 
way  to  be  able  to  also  adopt,  besides  to  the  two  basic 
function  (DGPS  present,  DGPS  absent),  some  other 
operating  characteristics,  in  order  to  make  it  possible 
to  take  into  account  the  characteristics  of  the  DGPS 
signal:  in  fact,  as  it  has  been  previously  anticipated, 
the  parameter  that  allows  to  commute  among  the 
different  modules  is  PDOP,  that  is  the  Position 
Dilution  of  Precision. 

The  architecture  of  the  Kalman  Filter  will  then  be 
able  to  commute  among  different  operating 
characteristics,  depending  on  the  PDOP  value,  that 
will  become  the  control  variable  from  which  the 
different  integration  strategies  will  be  applied. 

In  particular,  PDOP  is  able  to  point  out  if  the  measure 
of  the  vehicle  position  calculated  by  DGPS  is  reliable 
or  not:  if  PDOP  is  low'  (<3),  data  coming  from  DGPS 
have  to  be  considered  as  reliable,  while,  if  PDOP  has 
an  average  value  (e[3,6])  or  an  high  (>6)  value, 
DGPS  data  will  have  a  low  reliability  and  this  fact 
will  influence  the  trustfulness  of  the  measures  and  the 
trustfulness  of  the  estimations. 

In  order  to  take  into  account  of  the  PDOP  value,  and 
then  of  the  error  on  the  DGPS  measures,  the 
increasing  error  on  the  measures  is  automatically 
’  reflected  on  the  increasing  of  the  variance  of  the 
noise  corrupting  the  status  vector. 
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In  particular,  the  adopted  Multi-hypothesis  strategy 
allows  us  to  take  into  account  the  variability  of  the 
corrupting  noise  by  adaptively  modifying  the 
covariance  matrices  of  the  error  on  the  observations 
and  on  the  status,  depending  on  the  PDOP 
characteristics. 

In  such  a  war’,  the  KF  is  able  to  switch  among 
different  integration  strategies,  depending  on  the 
PDOP  current  value. 

The  variance  of  the  observation  is  run-time  changed 
according  to  this  value:  if  the  PDOP  is  low  (<4) 
DGPS  data  are  considered  reliable  and  the  variance 
of  the  observation  is  small. 

This  variance  increases  as  the  PDOP  increases  (from 
middle  (e  [4,  8])  to  high  (>  8)  values).  As  the 
variance  of  the  measure  increases,  the  reliability  in 
current  observations  is  reduced. 

In  order  to  do  that,  the  different  observ  ation 
variances,  applied  as  the  PDOP  varies,  are  assessed  in 
a  preliminary  off-line  phase  on  the  basis  of  a  suitable 
training  set  of  data. 

In  particular,  the  different  variances  that  are  applied 
to  the  system  measurements  have  previously  been 
computed  on  an  experimental  basis,  through  a  data 
training  set  in  which  PDOP,  DGPS  measures  and 
corresponding  real  measures  are  known. 


The  developed  integration  system  will  be  able  to 
follow  the  vehicle  even  in  the  worst  case,  that  is  full 
absence  of  the  DGPS  signal. 

Through  the  proposed  approach,  even  if  the  DGPS  is 
missing,  the  system  can  track  the  vehicle  by  using  the 
status  vector  obtained  through  the  subtraction  of  the 
estimated  cinematic  variables  from  the  current  values 
of  the  INS. 

Being  the  tight  integration  strategy  implemented  in  a 
closed  loop  configuration,  position,  speed  and 
acceleration  that  have  been  estimated  will  become  the 
feedback  of  the  system. 

11.  EXPERIMENTAL  RESULTS 

In  this  paragraph,  the  results  obtained  through  the 
proposed  integration  between  DGPS  and  INS  data  are 
proposed. 

The  system  has  been  tested  on  an  extensive  set  of  real 
data  acquired  din  ing  many  vehicle  missions  on  urban 
and  suburban  roads  in  North  Italy. 

The  results  of  the  DGPS/INS  integration  system 
proposed  (Fig.  3)  derive  from  the  data  acquired 
during  a  test  mission  in  which  DGPS  data  w  ere 
missing  for  some  periods  (Fig.  2). 


Figure  2.  Example  oj  DGPS  data  related  to  a  5  minute  long  mission.  The  DGPS  data  are  missing  four  times  (where 

spots  are  absent). 


As  it  can  be  observed,  the  proposed  system  is  able  to 
restore  the  trajectory  of  the  vehicle,  by  covering 
through  the  INS  data  the  loss  of  the  DGPS  data:  this 
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integration  allows  one  to  have  the  complete  trajectory 
of  the  vehicle  even  when  the  DGPS  signal  is  missing. 


Figure  3.  The  result  of  the  DGPS/INS  integration  referring  to  the  mission  in  Fig.  2:  the  trajectory  of  the  vehicle  is 

restored. 


The  per-cent  error  statistics  of  the  test  mission  are 
shown  in  T able  1 .  The  error  was  computed  as  the 


absolute  Euclidean  Distance  between  the  reference 
and  the  estimated  trajectory.  The  sample  percentage 
and  the  mean  error  are  reported. 
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Error 

North 

East 

(mt) 

% 

Mean 

% 

Mean 

<2 

65 

0.48 

91 

0.58 

[2,  5) 

33 

3.11 

7 

2.47 

[5,  10) 

2 

5.98 

2 

5.43 

>10 

0 

0 

0 

0 

Table  1  Per-cent  error  statistics  of  the  restored  trajectory. 


Also  these  quantitative  figures  show  that  the  error 
made  in  the  position  estimation  trajectory  remains 
quite  small  even  in  the  cases  when  DGPS  signal  is 
absent. 

CONCLUSIONS 

In  the  present  w  ork  the  problem  of  the  continuous 
estimation  of  the  position  of  a  vehicle  in  the  ground 
has  been  addressed. 

The  data  that  have  been  used  in  order  to  deifne  the 
current  position  of  the  vehicle  and  its  displacements 
are'  a  Different  Global  Positioning  System  (DGPS) 
and  an  Inertial  Navigation  System  (INS). 

The  optimal  estimation  of  the  vehicle  position  has 
been  obtained  through  the  integration  and  fusion  of 
the  information  coming  form  each  one  of  the  used 
sensors.  The  approach  followed  for  the  integration  is 
a  KF-based  Multi-Hypothesis  Approach. 

The  proposed  solution  has  been  shown  able  to 
provide  a  robust  system  able  to  compute  in  a  precise 
way  the  position  of  the  vehicle  and  resistant  to 
corrupting  noise.  Experimental  tests  on  the  field  have 
shown  the  goodness  of  the  results  obtained  through 
the  proposed  algorithm. 
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PAPER  No:  16 

DISCUSSOR’S  NAME:  R.  Sabatini 

COMMENT/QUESTION: 

Do  you  think  that  your  system  can  be  used  successfully  for  Airport  Traffic  Management  (on  the 
ground)? 

Do  you  think  that  a  DGPS/Radar  approach  would  be  more  suited  for  this  application? 


AUTHOR/PRESENTER’S  REPLY: 

Yes,  the  system  can  be  used  for  tracking  any  vehicles  on  the  ground,  but  due  to  integrity  issues 
(related  to  DGPS)  and  considering  that  angular  data  are  not  strictly  required  for  airport  traffic 
management,  I  think  the  DGPS/Radar  approach  would  be  more  suitable. 


DISCUSSOR’S  NAME:  R.  Sabatini 

COMMENT/QUESTION: 

Are  you  thinking  about  implementing  “carrier  phase"  DGPS  techniques  together  or  an  alternative 
to  “pseudo-range”  techniques? 


AUTHOR/PRESENTER’S  REPLY: 

Not  at  the  moment.  However,  an  upgrade  of  the  system  to  interpherometric  DGPS  techniques  is 
foreseen. 


DISCUSSOR’S  NAME:  R.  Sabatini 

COMMENT/QUESTION: 

Which  kind  of  integration  scheme  have  you  adopted? 


AUTHOR/PRESENTER’S  REPLY: 

Tightly-coupled  DGPS/INS.  In  particular,  a  closed-loop  integrated  DGPS/INS  architecture. 
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PAPER  No:  16 

DISCUSSOR’S  NAME:  R.  Sabatini 

COMMENT/QUESTION: 

Does  the  system  process  attitude  information  (i.e.  angular  data  from  the  INS)? 
Reply: 

No.  only  position  is  computed. 

Which  kind  of  INS  are  you  using  (strapdown/platform)? 

Reply: 

Strapdown 


DISCUSSOR’S  NAME:  G.  S.  Brown 

COMMENT/QUESTION: 

Do  you  have  data  that  verifies  that  your  noise  is,  indeed,  Gaussian? 


AUTHOR/PRESENTER’S  REPLY: 

In  order  to  verify  the  Gaussian  assumption  made  on  the  experimental  data  used,  in  particular 
concerning  the  dynamic  behaviour  of  the  error  introduced  by  inertial  sensors  in  speed 
computation,  a  test  for  Gaussian  (Kolmogorov  -  Smirnov)  has  been  performed. 

In  particular,  it  has  confirmed  the  validity  of  the  assumption 

£>  ~  J'  [  K  )  I  f 

CL  '  c.  J 


shown  during  the  presentation. 
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Summary 

ARTAS  (an  acronym  for  ATC  Radar  T racker  and  Server)  is  currently  in  pre-operational  test  at  four 
different  sites  in  France,  Germany  and  the  Netherlands.  The  ARTAS  system  consists  of  a  tracker, 
responsible  for  maintaining  up-to-date  target  state  vectors,  and  a  server,  which  handles  client 
subscriptions  (e.g.  from  the  ATC  display  system)  and  delivers  the  target  state  vectors  to  these 
clients.  An  ARTAS  system  co-operates  with  adjacent  ARTAS  systems  by  exchanging  target  state 
vector  information. 

The  main  features  of  the  ARTAS  T racker  are 

•  tracking  with  up  to  thirty  radars  (PR,  SSR  or  CMB) 

•  on-line  estimation  of  the  radar  systematic  errors 

•  on-line  estimation  of  the  radar  accuracy  and  coverage 

•  high-accuracy  position  and  velocity-vector  estimation 

•  responsiveness  to  target  manoeuvres 

•  insensitivity  to  clutter 

•  target  classification 

The  tracking  filters  are  interacting  multiple-model  (IMM)-based  filters,  a  four-model  filter  for  high¬ 
speed  and  highly  manoeuvring  targets  and  a  two-model  filter  for  low-speed  targets  [1],  The  plot-to- 
track  association  is  based  on  probabilistic  data  association  (PDA),  with  special  joint  probabilistic 
data  association  (JPDA)  algorithms  in  case  of  target  close  approach  situations  [2].  Track  initiation  is 
done  by  time-reversed  multiple-hypothesis  tracking.  Target  classification  is  based  on  Shafer- 
Dempster  reasoning. 

Introduction 

ARTAS  is  designed  as  a  track  data  server.  Track  data  users  can  subscribe  to  a  certain  service  and 
receive  the  track  data  in  ASTERIX  format  via  a  local-area  or  wide-area  network  (LAN/WAN,  figure 
1).  Users  can  be  ATC  centres,  flightplan  processing  systems,  air-traffic  management  units  and  so 
on.  Each  user  can  have  a  dedicated  service,  taking  into  account  requirements  with  respect  to  data 
contents  and  update  frequency.  An  ARTAS  unit  also  receives  its  input  data  from  the  radars  via  the 
local-area  or  wide-area  network.  Furthermore,  an  ARTAS  unit  can  communicate  via  the  network 
with  other,  adjacent,  ARTAS  units  in  order  to  provide  a  continuous  air-picture  to  its  users.  T rack 
data  from  adjacent  units  is  used  to  accelerate  the  initiation  of  tracks  at  the  border  of  the  unit’s  own 
domain  of  interest  (DOI)  and  to  smooth  the  transition  of  a  track  from  one  unit’s  DOI  to  another  unit’s 
DOI.  Finally,  when  there  is  sufficient  coverage  of  the  own  unit’s  DOI  by  adjacent  ARTAS  units,  the 
adjacent  ARTAS  units  can  take  over  the  surveillance  in  case  of  an  own  unit  failure.  Thus, 
enhancing  the  overall  reliability  of  the  surveillance. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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The  internal  structure  of  an  ART AS  unit  is  shown  in  figure  2.  The  Router  Bridge  is  the  interface  to 
the  external  network.  It  pre-processes  the  incoming  radar  data,  i.e.  it  performs  format  checks  and 
sectorisation  of  the  plot  data  and  keeps  track  of  the  operational  status  of  the  radars.  The  Server  is 
responsible  for  the  handling  of  ARTAS  user  requests  and  the  distribution  of  the  track  data, 
according  to  the  different  user  services.  The  most  simple  service  that  is  provided  is  a  regular 
broadcast  of  all  track  data.  MMI/Supervision  is  the  man-machine  interface  and  supervision  unit.  It 
provides  a  basic  display  of  the  unit  tracks  and  control  functions  for  the  ARTAS  unit.  The  Tracker, 
finally,  is  responsible  for  keeping  an  up-to-date  air  picture.  An  ARTAS  unit  consists  of  two  identical 
chains  of  a  Router  Bridge/Tracker/Server/MM I/Supervision  subunits.  All  subunits  operate  in  a 
multiple-computation  redundancy  mode;  that  is,  there  is  a  master  and  a  slave  subunit  that  both 
perform  the  same  processing,  except  that  the  slave  subunit  does  not  provide  any  output.  Instead, 
the  slave  performs  some  additional  processing  to  keep  master  and  slave  in  synchronisation. 


All  the  ARTAS  subunits  run  on  off-the-shelf  hardware  and  are  programmed  in  ADA,  except  for  the 
MMI,  which  is  programmed  in  C++. 

The  ARTAS  Tracker 

Basically,  the  task  of  the  tracker  is  to  provide  estimates  of  the  state  of  aircraft  in  the  domain  of 
interest  of  the  ARTAS  unit.  It  makes  use  of  maximum  30  sensors;  present  types  are  primary  radar 
(PR)  and  secondary  surveillance  radar  (SSR).  Extensions  to  incorporate  Mode-S  and  automatic 
dependent  surveillance  (ADS)  data  are  foreseen  for  1 998. 


A  prime  requirement  for  handling  multisensor  data  is  the  ability  to  cope  with  sensor  alignment 
errors,  i.e.  systematic  radar  errors  like  position  bias,  range-  and  azimuth  bias,  but  also  time- 
stamping  bias  and  transponder-delay  error.  The  latter  is  an  example  of  a,  so-called,  micro-error;  a 
systematic  error  that  depends  on  the  object  being  tracked.  The  former  errors  are  macro-errors;  they 
only  depend  on  the  sensor  involved.  Unfortunately,  systematic  errors  may  change  in  time,  due  to 
e.g.  changing  atmospheric  conditions  and  radar  maintenance.  Therefore,  the  ARTAS  Tracker 
contains  an  on-line  systematic-error  estimation  module  that  is  able  to  track  varying  systematic  radar 
errors. 

Another  requirement  for  handling  multisensor  data  is  a  proper  treatment  of  coordinate 
transformations.  This  becomes  a  more  obvious  problem  when  the  size  of  the  system  area 
becomes  large.  ARTAS  uses  WGS84  as  a  reference  system.  Measurement  processing  and  track 
update  processing  are  done  in  local  Cartesian  systems,  such  that  the  error,  induced  by  coordinate 
transformations,  is  minimised.  This  implies  that  all  sensors  and  all  tracked  objects  have  their  own 
local  Cartesian  system  that  may  change  in  time  when  objects  move. 

Track  continuation  uses  the  reports  of  all  available  sensors  to  estimate  the  state  of  a  target.  Each 
track  extrapolation/update  cycle  is  based  on  the  reports  of  a  single  sensor,  though.  Subsequent 
cycles,  however,  may  be  of  entirely  different  sensors.  Prior  to  the  track  update,  all  the  relevant 
reports  are  corrected  for  micro-errors  (systematic  errors  that  vary  from  target  to  target)  and  slant- 
range  effects.  Track  continuation  is  discussed  in  more  detail  below. 

Track  initiation  is  done  based  on  the  reports  of  single  sensors  only.  It  is  based  on  multiple- 
hypothesis  tracking  (MHT)  and  is  done  retrospectively  [3].  Considering  the  fact  that  a  new  target 
generally  enters  the  coverage  of  only  a  single  radar,  the  gain  of  a  shorter  track  initiation  delay  did 
not  warrant  the  additional  complexity  of  a  multi-radar  initiation  in  a  civil  ATC  environment.  This 
trade-off  may  not  be  valid  in  a  military  environment,  though. 

The  ARTAS  T racker  maintains  aircraft  and  non-aircraft  tracks  since,  in  many  cases,  the  best  way  of 
dealing  with  anomalies,  like  reflections  and  sidelobes,  is  to  track  them  and  to  classify  them  as  being 
non-aircraft.  To  that  end,  the  ARTAS  Tracker  contains  a  track  classification  module,  which 
classifies  tracks  using  Shafer-Dempster  reasoning  [4].  The  criteria,  used  in  the  classification,  are 
based  on  radar  environment  characteristics,  target  behaviour  and  a  set  of  models  for  specific 
anomalies,  like  reflections  and  sidelobes.  An  advantage  of  Shafer-Dempster  reasoning  is  the  ease 
with  which  additional  criteria,  like  target  signature  information,  can  be  incorporated  into  the 
classification  process. 

Track  Continuation 

For  the  ARTAS  T racker,  a  Bayesian  approach  to  track  continuation  was  adopted.  This  approach 
did  prove  to  yield  a  high-performance  tracker,  as  experience  with  the  NLR  JUMPDIF  prototype 
tracker  has  shown  [1]. 

Basically,  there  are  four  major  problems  that  occur  during  track  continuation 

1 .  Non-linear  aircraft  dynamics  during  a  turn 

2.  The  association  of  measurements  with  existing  tracks 

3.  The  occurrence  of  outlier  measurements  (non-Gaussian  measurement  noise) 

4.  Sudden  starts  and  stops  of  manoeuvres 

For  each  of  these  problems,  adequate  solutions  were  already  developed  for  the  JUMPDIF 
prototype  [1];  the  result,  an  Interacting  Multiple-Model  Probabilistic  Data-Association  (IMMPDA) 
algorithm  with  Extended  Kalman  Filters  (EKF)  [6]  was  used  in  extensive  performance  tests.  The 
results  of  these  performance  tests  were  used  as  a  basis  for  the  ARTAS  T racker  performance 
requirement  specification.  A  number  of  improvements,  with  respect  to  the  JUMPDIF  tracker,  were 
made  in  the  ARTAS  T racker,  though. 

For  target  resolution  situations,  new  joint  probabilistic  data-association  (JPDA)  algorithms  were 
developed  [2]  that  perform  considerably  better  than  the  probabilistic  data-association  (PDA) 
algorithm,  that  is  used  in  JUMPDIF.  These  JPDA  algorithms,  however,  require  more  computations 
than  the  PDA  algorithm.  In  order  to  save  CPU-load,  these  JPDA  algorithms  are  only  used  when  a 
target  resolution  situation  is  detected. 

The  ARTAS  Tracker  is  required  to  track  targets  down  to  zero  groundspeed.  In  general,  it  is  not 
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necessary  to  track  iow-groundspeed  targets  with  an  advanced  four-model  (left  turn,  right  turn, 
change  of  groundspeed,  straight  flight)  IMMPDA  filter  to  get  a  good  tracking  performance. 
Therefore,  a  simplified  two-model  (manoeuvring  flight,  straight  flight)  IMMPDA  filter  is  used  to  track 
these  targets. 

JUMPDIF  contained  a  two-model  (climb/descent,  level  flight)  IMMPDA  filter  for  SSR  mode-C 
measurements.  In  the  ART AS  Tracker  this  filter  was  replaced  by  a  three-model  (climb,  descent, 
level  flight)  IMMPDA  filter  in  order  to  be  more  responsive  to  changes  in  the  rate  of  climb/descent. 
Furthermore,  two  algorithms  to  estimate  the  target  altitude  in  absence  of  SSR  mode-C  information 
were  implemented.  One  algorithm,  Triangulation,  is  discussed  in  more  detail  below.  The  other 
algorithm,  Height-from-Coverage,  uses  the  assessed  coverage  of  each  radar,  that  detects  or  does 
not  detect  the  target,  to  calculate  a  height  interval  for  the  target.  Although  not  very  accurate,  using 
the  result  of  this  algorithm  is  often  better  than  using  a  default  altitude. 


NLR  -  JDiff:  Example  of  radar  bias  Wed  Sep  24  14:06:28  1997 


Nautical  miles  Trackld:  313 


dbltrk  track_data  09:48:59.4  -  09:55:00.0 


Figure  3.  Example  of  biased  plots 

The  key  solution  to  the  multisensor  track  continuation  problem  that  is  applied  in  the  ART  AS  Tracker 
is  a  proper  correction  for  the  macro-  and  micro-systematic  errors  of  all  involved  measurements, 
before  they  are  used  within  the  track  extrapolation/track  update  cycle.  This  essentially  reduces  the 
multisensor  problem  to  a  single-sensor  problem.  The  time  sequence  of  track  extrapoiation/track 
update  cycles,  obviously,  contains  track  extrapolation/track  update  cycles  for  all  the  available 
sensors.  The  difference  between  cycles  for  different  sensors  is  the  use  of  a  different  measurement 
matrix  for  the  Extended  Kalman  filters. 

Figure  3  shows  a  track,  departing  from  Schiphol  airport,  that  uses  biased  measurements  from  three 
different  radars.  Triangles  indicate  the  raw  plots,  crosses  the  nearest-neighbour  plot  positions 


(corrected  for  the  estimated  radar  biases)  and  the  squares  the  updated  track  position.  The  vectors 
indicate  the  predicted  flightpath  up  to  the  next  measurement  instant.  Figures  4  and  5  show  the 
ART AS  Tracker  estimates  of  the  groundspeed  and  SSR  mode-C  height  of  this  track,  respectively. 


NLR  -  JDiff:  track  groundspeed  Wed  Sep  24  14:09:02  1997 
Groundspeed 
Track:  313 
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NLR -JDiff:  track  height  Wed  Sep  24  14:10:21  1997 

Height 
Track:  313 


Figure  4.  Track  groundspeed  estimate  Figure  5.  Track  mode-C  height  estimate 

Systematic  Radar-Error  Estimation 

The  ARTAS  Tracker  estimates  the  following  (macro-)systematic  errors: 

•  range  bias 

•  azimuth  bias 

•  range  gain  (a  range  bias  that  increases  with  increasing  range) 

•  antenna  squint  (non-verticality  of  the  plane  of  the  radar  beam) 

•  verticality  error  (antenna  rotation  axis  not  perpendicular) 

•  time-stamping  bias 

The  problem  with  dynamic  estimation  of  the  (macro-)  systematic  errors  is  that,  in  principle,  the  filter 
equations  are  coupled  with  the  track  continuation  equations  of  the  individual  tracks.  It  is,  of  course, 
very  well  possible  to  make  a  selection  of  a  small  number  of  well-behaved  tracks  and  to  solve  the 
resulting  set  of  equations.  In  [5],  a  different  approach  is  taken,  which  decouples  the  equations  for 
(macro-)systematic  error  estimation  from  the  track  continuation  equations.  Effectively,  it  comes 
down  to  integration  of  the  innovations  of  all  tracks  and  filtering  these  innovations  with  a  Kalman 
filter.  Due  to  the  larger  timeconstant  of  the  systematic  error  process,  the  filtering  equations  become 
independent  of  the  individual  track  maintenance  equations.  This  algorithm  is  implemented  in  the 
ARTAS  T racker  and  uses  a  selection  of  non-manoeuvring  tracks  in  order  to  save  CPU-load  and  to 
increase  the  speed  of  convergence  of  the  estimation  process.  Figures  6  and  7  show  results  of  the 
(macro-)  systematic-error  estimation  process  on  a  2-radar  PR  scenario. 
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NLR-JDiff:  Thu  Sep  25  09:41:08  1997 


NLR-JDiff:  Thu  Sep  25  09:42:46  1997 


Range  Bias  for 

— =  Primary  value 
=  Secondary  value 


Radarld:  1 

=  Primary  standard  deviation 
-  Secondary  standard  deviation 


Range  Bias 

=  Primary  value 
— =  Secondary  value 


for  Radarld:  5 

"•=  Primary  standard  deviation 
•"=  Secondary  standard  deviation 


log2_raw_plts_prep.rndr  sys_err_5  15:15:00.0- 15:45:00.9 


log2_raw_p!ts_prep.mdr  sys_err_5  15:15:00.0-  15:45:00.9 


Figure  6.  TAR  estimated  range  bias  Figure  7.  LAR  estimated  range  bias 

Triangulation 

After  estimation  of  the  systematic  radar  errors  that  are  radar-dependent  only  (macro  errors),  the 
track-related  errors  (micro  errors)  can  be  estimated.  Within  the  ARTAS  Tracker,  these  micro  errors 
consist  of  the  transponder  delay  error  (i.e.  the  difference  between  the  actual  delay  and  the  nominal 
value  of  3  microsecond  as  specified  by  ICAO)  and  the  geometric  height,  estimated  from  range- 
azimuth  position  measurements  in  a  multi-radar  environment. 

A  general  solution  to  this  problem  is  to  extend  the  state  vector  of  an  object  with  these  components 
and  to  extend  the  corresponding  extended  Kalman  filter  equations  accordingly.  Since  this  is  a  very 
costly  solution  (in  terms  of  CPU),  we  have  looked  for  a  robust  method  that  is  not  coupled  with  the 
track  continuation  equations.  In  situations  where  an  SSR  radar  has  a  co-located  primary  radar,  a 
robust  method  to  estimate  the  transponder  delay  error  is  to  average  the  difference  in  range 
measurements  of  the  two  radars.  In  other  situations,  the  transponder  delay  error  and  geometric 
height  estimations  are  coupled. 

Consider  the  situation  that  two  non-co-located  radars  observe  an  object  at  the  same  moment  in 
time.  To  perform  triangulation,  we  use  the  difference  between  the  projections  of  the  plots  to  a 
common  2-dimensional  Cartesian  coordinate  system  (the  track-local  coordinate  system)  as  the 
innovation  term  in  a  Kalman-like  filter  update  step  for  the  estimation  of  the  transponder  delay  error 
and  the  geometric  height. 


Since  a  simultaneous  measurement  of  one  object  by  two  non-co-located  radars  is  quite  unusual, 
we  perform  a  triangulation  on  the  basis  of  a  triplet  of  projected  plot  positions  (under  the  condition’ 
that  the  track  groundspeed  and  course  are  constant):  the  first  and  third  projected  position  are 
interpolated  to  the  time  of  the  middle  plot. 

The  performance  of  this  algorithm  depends,  among  others,  on  the  geometric  configuration  of  the 
radars  involved:  the  middle  plot  should  be  from  a  different  radar  than  the  other  two  plots,  with  a  line- 
of-sight  opposite  to  that  of  the  other  radars,  and  as  close  to  the  object  as  possible. 

In  figure  8,  we  see  a  part  of  a  track  from  a  live  data  collection.  The  recording  was  made  for  3 
secondary  and  2  primary  radars,  but  the  Tracker  was  run  with  only  the  primary  plot  data.  The  track 
is  flying  at  FL  290  (8839.2  m);  the  plots  are  not  corrected  for  systematic  radar  errors.  The  estimate 
of  the  geometric  height  and  the  1 -sigma  margin  are  given  in  figure  9;  the  initial  estimate  is  6000  m. 


NLR  -  JDiff:  track  at  FL  290  Tue  Sep  23  15:47:39  1997 


NLR  -  JDiff:  Triang.  height  Tue  Sep  23  14:46:33  1997 


Z  Estimate 
Track:  11 


log2_raw_plts_prep.mdr  track_data  IS:  15:00.0-  15:25:00.0 


Figure  8.  Track  observed  by  2  PR  radars  Figure  9.  Triangulated  height  as  function  of  time 

Conclusion 

Adequate  systematic  error  estimation  is  a  pre-requisite  for  accurate  multisensor  tracking.  In  the 
ART AS  Tracker,  several  powerful  methods  are  employed  for  the  on-line  estimation  of  both  macro- 
and  micro-systematic  errors.  These  methods  provide  accurate  estimates  of  the  systematic  errors 
as  shown  by  a  number  of  examples.  By  having  accurate  systematic  error  estimates,  the 
multisensor  problem  is  essentially  reduced  to  a  time-sequential  single-sensor  problem,  which  is, 
obviously,  much  easier  to  solve. 
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PAPER  No:  17 

DISCUSSOR’S  NAME:  P.  Hoogeboom 
COMMENT/QUESTION: 

The  desired  accuracy  you  mentioned  is  1 1  meters.  The  aircraft  you  measure  are  much  larger, 
and  can  show  glints.  How  does  this  influence  the  accuracy  and  the  correction  schemes? 


AUTHOR/PRESENTER’S  REPLY: 

The  radar  measurements  presented  to  the  system  are  point  measurements,  probably  based  on 
the  centroid  of  the  reflected  radar  image.  So  the  estimator  cannot  take  into  account  the  physical 
extent  of  the  target.  On  the  other  hand,  the  estimated  bias  is  based  on  the  measurement  of  a 
large  number  of  targets,  both  close  by  and  far  away,  so  the  physical  extent  and  the  glints, 
resulting  from  that,  do  not  affect  the  estimator  very  much. 


DISCUSSOR’S  NAME:  M.  Desbois 


COMMENT/QUESTION: 

Does  the  integration  of  mode  S  or  GPS-equipped  aircraft  modify  your  approach  concerning  the 
selection  of  tracks  for  the  debiassing. 

AUTHOR/PRESENTER’S  REPLY: 

The  method  selected  for  ARTAS  remains  mainly  on  sensor  selection  and  current  traffic  selection 
based  on  several  criteria,  among  with  the  quality  of  tracks  that  takes  into  account  mode  S  or  GPS 
information. 


Low  Altitude  Wave  Propagation  Effects  in  a 
Transmitter  Independent  Receiver  Network. 
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Abstract: 

For  the  design  and  implementation  of  a  Passive 
Multistatic  Early  Warning  and  /  or  Fire  Control 
Radar  (T.I.R.N.,  Transmitter  Independent 
Receiver  Network,  [1])  many  aspects  include  the 
environmental  conditions  that  T.I.R.N’s  may 
encounter  at  operational  conditions. 

Even  operating  in  a  standard  troposphere,  the 
result  of  the  refraction  phenomenon  is  the  curved 
propagation  of  the  electromagnetic  waves.  The 
problem  becomes  more  complicated  at  low 
altitudes,  where  tropospheric  refractive  index 
variability,  multipath  propagation  and  clutter 
(sea  or  ground)  must  be  taken  into  consideration 
in  order  to  avoid  erroneous  results  for  the 
location  of  a  target  as  well  as  its  velocity 
estimation.  Although  simplified  models  of  the 
above  environmental  parameters  are  used  in  this 
analysis,  (e.g.  lateral  refraction  variability  is  not 
studied  here,  and  Weibull  Distribution  Model  is 
taken  for  granted  as  regards  the  sea  clutter), 
useful  results  can  be  produced  for  the  evaluation 
of  multistatic  systems. 

In  this  paper  a  four-receiver  model  using  the 
signal  of  a  non-cooperative  transmitter  is 
evaluated  for  its  sensitivity  at  the  above 
conditions  and  is  compared  with  a  conventional 
monostatic  radar  operating  with  the  same  signal. 
It  is  shown  that  the  T.I.R.N.  proposed  is  a  more 
efficient  system  than  its  monostatic  equivalent. 
The  results  presented,  despite  of  being  strictly 
theoretical  (not  yet  evaluated  in  real  operational 
conditions)  may  help  the  design  of  multistatic 
radars. 

1.  Introduction. 


As  it  has  been  discussed  in  [1]  T.I.R.N.S 
operation  depends  strongly  on  the  spatial 
separation  of  the  receivers.  Actually  it  is  shown 
[2]  that  for  any  target  case  and  mode  of  location 
(Triangulation,  T.D.O.A.,  or  Range  Sum)  on  a 
4-receiver  model,  the  receivers  must  not  be 
located  at  the  same  planar  surface  for  3-D 
location  and  dopller  vector  velocity 
measurement.  (Later  in  this  paper  this  constraint 
will  be  strengthened.) 

That  means  that  receivers  must  be  placed  in 
different  altitudes  also.  So  a  realistic  model 
includes  receivers  in  an  altitude  range  of  30  to  a 
mere  2000  meters. 

Also  we  must  note  that  in  this  paper  the  radar 
transmitter  is  non-cooperative  it  is  located  in  an 
ambiguous  location,  and  is  using  an  unknown 
antenna  pattern  of  radiation.  One  of  main 
purposes  of  T.I.R.N.  design  is  the  ability  to  use 
enemy  radars  as  target  illuminators. 

2.  Model  Consideration. 

It  is  understandable  that  if  the  transmitter  and/or 
the  target  is  located  at  a  low  altitude  then 
tropospheric  effects  must  be  considered  in  order 
to  avoid  erroneous  results.  In  this  paper  the 
model  considered  is  consisted  of  four  receivers: 
Two  at  an  altitude  of  less  than  100  m  ,  one  at  an 
altitude  of  about  500m  and  one  at  altitude  of 
about  1500m.  The  distances  between  the 
receivers  vary  from  10  to  40  km.  (Fig.l)  It  is 
assumed  that  for  adequate  triangulation  location 
the  receivers  are  equipped  with  2  or  4  element 
monopulse  antennas.  The  reason  of  using 
monopulse  techniques  is  that  they  give  good 
angular  discrimination  even  retaining  wide 
beamwidths  for  best  bistatic  coverage.  Their 
elements  are  assumed  to  have  wide  horizontal 
and  narrow  vertical  beamwidths.  (multiple  horn 
antennas  or  interlaced  phased  arrays  are 
considered ). 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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For  this  configuration  a  useful  T.I.R.N.  range  is 
about  120  km. 


B 


Fig  I. Two-  Beam  Monopulse  antenna  patterns.  Targert  is  in  boresight 
level  when  signal  strengths  at  A,  B  are  equal.  4-element  can  be  used 
for  both  azimuth  and  elevation  discrimination. 

Then  we  can  consider  cartesian  coordinate 
systems  attached  to  the  receivers  all  aligned  to  a 
central  “point  of  interest”  coordinate  system. 
That  point  of  interest  could  be  any  receiver,  a 
command  post  of  a  weapon  system  in  the  area.  In 
modest  latitudes  (x,y,z)  can  be  North,  West, 
Altitude  corrected  by  first  order  approximation 
for  the  mean  radius  of  the  earth  R=6378km.  That 
means  that  altitudes  must  be  corrected  by  the 
factor  d2/R  and  the  angles  between  the  axis  by  an 
order  of  180d/7iR  degrees  or  about  d/n  mils  if  R,d 
in  km.  More  accurately  the  axes  must  be 
corrected  by  the  exact  differences  of  Latittude 
and  Longitude  of  the  receiver  and  the  “point  of 
interest”. 

The  main  purpose  of  this  paper  is  to  correct 
radar  measurements  to  real  atmospheric 
conditions  at  least  as  they  are  theoretically 
modeled . 

2.1.  Some  Free  Space  Calculations. 

If  transmission,  scattering  and  reception  were 
taking  place  to  free  space  then  the  location  and 
the  velocity  measurements  would  require  simple 
algebraic  calculations.  The  corrections 
mentioned  earlier  can  be  made  during 
installation.  When  using  triangulation  method 
the  location  of  the  target  or  transmitter  can  be 
determined  by  the  following  set  of  calculations: 

If  <pi  and  cp2  are  the  look  angles  (azimuths)  and 
(x^yi)  (x2,y2)  are  the  receiver  locations  then  the 
transmitter  location  (x,y)  is  extracted  by  the 
linear  system: 


COS^Jj 

sin^j 

X 

x,cos  9,  +>'  sin  9, 

cos<p2 

sin92 

y. 

x2  cos  9,  +y2  sin92 

And  for  z 
2  =  zj  +  d  cot  i?  j 

d  =  jix-X')1  +(y-y,)2 

(2) 


In  this  case  0  is  the  measured  from  the  z  axis  of 
the  ith  transmitter.  The  reason  z  (altitude)  is 
separeted  will  be  discussed  later  in  this  paper. 

A  serious  shortcoming  of  the  triangulation 
method  is  the  target  “ghosts”  that  can  be 
presented  if  there  are  more  than  one  targets  in 
the  area.  So  additional  method  of  location  is 
required  for  target  location  verification  and  this 
can  be  T.D.O.A.  or  Range  Sum  Method 

(2) , [3], [4]  if  the  target  is  close  or  remote  to  the 
T.I.R.N.  area  respectively.  Range  Sum  is  also 
desirable  for  a  single  receiver  target  location. 
These  methods  are  generally  non-linear  and  so 
computing  complexity'  makes  them  more  slow. 

An  interesting  feature  of  T.I.R.N’s  is  the  velocity 
vector  estimation  by  measuring  (where  the  signal 
is  appropriate)  the  Doppler  shift  at  each  receiver. 
It  is  known  that : 

f  =  _  fr(V,  +  Vr) 

J  D 

C 

(3) 

where  fD  and  ft  are  the  Doppler  frequency  shift 
and  the  transmitted  frequency'  respectively,  vt 
and  vr  are  the  radial  velocities  of  the  target 
relative  to  the  transmitter  and  the  receiver.  Note 
that  they  are  the  signed  magnitudes  of  the 
projections  of  the  real  target  velocity  vector  on 
the  transmitter-target  (t-t)  and  receiver-target  (r- 
t)  directions. 

Using  a  four-receiver  T.I.R.N.  with 
synchronized  receivers  after  demodulation  we 
get  signals  with  frequencies  proportional  to 

S  =  Vi+Vn.+Ve 

(4) 

or 


y  ei  y  e 

(5) 

where  v,  is  the  q-t  signed  magnitude  (algebraic 
value)  of  the  velocity  vector  projection 
(1=1,2, 3,4),  ve  is  systematic  error  made  because 
of  vt  and  a  possible  frequency  demodulation  error 
-the  same  for  all  synchronized  receivers-,  vm 
noise  (random)  errors  which  can  be  taken  as 
uncorrelated  and  vei=  v,+Vni  is  the  estimation  of 
measurement  after  the  systematic  error  removal. 

For  the  velocity  vector  estimation  any  Cartesian 
or  spherical  target-centered  coordinate  system  is 
convenient.  If  v(u,w,s)  is  the  real  target  velocity 
vector  and  Vi  its  projection  in  a  random  direction 
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then: 

v  •  v  =  v2  y  —  12  3  4 

‘  1  1’Z’J’4  (6) 

Applying  equations  (4),  (5)  ,(6)  at  the 

coordinate  system  defined  we  get  for  the  r,-t 

directions  the  linear  following  system: 

wcos0  sin0  +wsin0i  sin6*  +s<xsdi  +v  =<5  / =1,2,34 

(7) 

where  (p,0  are  the  angular  coordinates  of  the 
receivers  relative  to  the  target  and  vni  is 
embodied  to  u,v,s. 

If  the  same  methodology  as  the  on-line  solution 
is  followed  then  this  is  equivalent  to: 

Gt>  =  5  u=[m  w  s  ve]T 


’sin0,  cos0, 

sin0j  cos0j 

cost?! 

1 

sin02  cos 

sin02  cos  02 

cos02 

1 

sin  03  cos  03 

sin  0,  cos  03 

COS  03 

1 

sin04  cos04 

sin04  cos  04 

COS04 

1 

s-iSASAf  (8) 

If  use  of  the  receiver  centered  coordination 
systems  is  desirable  then  all  the  cosines  of  the 
above  system  must  change  sign. 

Note  that  x,y,z,u,v,s  then  is  the  6-D  coordinates 
of  the  target  at  the  point  of  interest. 

Agtually  in  free-space  a  useful  G  could  be: 


X 

l 

x~ 

yi-y 

z,  -z 

r 

1 

y2~y 

z2  -z 

r 

r 

x3  -x 

y3~y 

z3  -z 

r 

X4  -x 

y4~y 

z  4 -Z 

r 

r  =  ^x2  +y2  +  z2 

(9) 

3.Standard  atmosphere  case. 

Another  simple  case  for  T.I.R.N.  environmental 
conditions  is  when  propagation  is  taking  place  in 
a  Standard  atmosphere  with  propagation  factor 
Kr4  (usually  K=4/3  but  can  raise  to  1.4  at 
Mediterranean  environments.)  In  that  case  all 
the  radiowave  measured  parameters  (distances, 
alltitudes,  and  elevation  angles  must  be 
corrected) 

The  correction  is  simple  and  straightforward  for 
z  assuming  that  propagation  factor  bends  the 


propagation  line  (ray)  towards  the  earth’s  surface 
then  equation  (3)  becomes: 


z  -  zi  =d  cot 


K- 1  d 2 
K  2  R 


(10) 


or 

K- 1  d2 

z  =  zfs - 

fs  K  2  R 

(11) 

where  z&  is  the  free  space  z  estimation. 


That  also  means  that  for  the  angle  of  departure 
of  this  ray: 


o  dz  a  K  +  l  d 

cot  t).  =  -  — —  =-cott?,+- 


dda=d 


K  R 


(12) 


From  equation  (10)  comes  the  strengthening  of 
the  condition  that  not  all  the  receivers  must  be  in 
the  same  level  plane.  If  Dmax  is  the  maximum 
distance  of  2  receivers  of  the  T.I.R.N.  then  a 
receiver  must  be  located  at  an  altitude  at  least 
order  of  magnitude  larger  than  [(K-l)Dmax2/KR] 
That  means  that  for  40  Km  receiver  separation  at 
least  a  pair  of  them  must  have  500  meters  of 
altitude  difference  or  higher  if  possible. 


Note  all  0’s  at  equation  (8)  are  actually  angles  of 
departure,  so  a  correction  must  be  done  to  them 
too,  in  order  to  avoid  erroneous  velocity 
estimation. 


It  must  also  be  noted  that  even  if  T.D.O.A.  or 
Range  Sum  is  used  for  target  location  the  same  z 
and  9  corrections  must  be  made.  This  is  because 
the  propagation  model  is  not  changing  by 
changing  the  method  of  location. 

4.Tropospheric  waveguide  (duct)  case 

In  environments  over  sea  evaporation  ducts  may 
occur 

at  very  low  altitudes  (less  than  30  m)  while  over 
ground  irregularities  of  the  air  temperature  and 
humidity  may  cause  irregular  refractive  index 
(or  modified  refractive  index  variability  able  to 
cause  waveguide  effects  at  altitudes  from  100  to 
300  m.  These  phenomena  are  more  severe  in 
environments  with  islands.  In  Mediterranean  sea 
there  are  waveguides  at  70%  of  time  in  the 
summer.  If  both  the  target  and  the  receiver  are  in 
the  waveguide  then  real  elevation  angle  data  are 
difficult  to  be  obtained. 

These  ducts  of  course  are  really  not  disturbing 
the  receivers  located  above  them  so  it  is  better  for 
the  T.I.R.N.  if  the  higher  placed  receivers  act  as 
height  finders  (z-finders  to  be  more  precise)  in 
order  to  avoid  evaporation  ducts.  The  opposite 


problem  is  taking  place  if  the  duct  is  taking  place 
over  ground.  Then  the  lowest  receivers  are  more 
likely  to  receive  direct  rays.  The  fact  that  only 
rays  with  elevation  angles  between  ±1.5  degrees 
are  waveguided  [5]  leads  to  a  simple  rule  that  is: 
Always  trust  the  larger  |0|  given  by  the  receivers 
and  if  all  |0|’s  are  smaller  than  1.5  degrees  trust 
the  highest  receiver.  That  rule 


Being  more  precise  in  that  case  equation  (1)  can 
still  be  used  (lateral  refractive  index  variabilities 
are  anyway  smaller  than  vertical),  but 
propagation  in  that  case  at  the  vertical  level 
cannot  be  confirmed  by  all  receivers  so  the  one 
with  the  less  altitude  discrimination  ambiguity 
can  be  used  and  corrected  as  described  at  (10). 
Bearing  then  in  mind  that  ranges  in  low  altitudes 
are  measured  in  tenths  of  Kilometers  and 
altitudes  in  hundrents  of  meters  then  a  good 
approach  in  velocity  estimation  can  be  done 
using  (10)  with  the  corrected  altitudes  and 
assuming  that : 

1  _  1  _  1 

r  d  d2  +  Z~  ambig. 

(13) 

Actually  the  above  approximation  can  be  used 
for  all  range  conversions  at  low  altitudes.  A 
simple  calculation  will  show  that  in  150  Km 
range  even  5Km  altitude  difference  would  give 
an  error  in  range  at  about  84  m  while  in  30  Km  , 
lKm  altitude  difference  would  give  a  range 
difference  of  17  meters  (or  the  length  of  a 
modem  fighter!)  with  that  simplifications  G  for 
an  approximate  velocity  estimation  can  be 
written  as: 
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5.  Multipath  propagation 

During  operation  at  low  altitudes  not  only  the 
direct  ray  but  also  the  reflected  ray  must  be  taken 
into  consideration.  The  fact  that  all  the  model 
consideration  has  been  done  with  unknown 
transmitter  location  one  useful  approximation  is 
to  consider  the  target  itself  a  secondary 
transmitter  and  so  the  problem  leads  to  a  point  to 
point  radiolink  problem.  [5], [6], 


Dealing  with  multipath  in  a  T.I.R.N.  having 
monopulse  antennas  however  the  problem 
becomes  more  complicated  because  the  reflected 
signal  undertakes  different  gains  at  the  lower 
and  upper  beams  of  the  antenna  or  their 
sidelobes.  That  would  give  significant  0  errors.  It 
has  been  stated  elsewhere  in  this  paper  that 
elevation  measurement  is  the  one  which  is  more 
sensitive  and  so  does  vertical  velocity  s  then.  An 
attempt  is  going  to  be  done  of  calculating  these 
effects: 


As  shown  in  Fig.2  a  monopulse  antenna  with 
beam  offset  angle  y  of  the  foresight  is  aiming  at  a 
target  at  an  angle  of  0=90  degrees  It  is  clear 
then  that  the  lower  beam  aims  at  0+y  while  the 
upper  beam  aims  at  0-y .  In  case  of  no  multipath 
then  the  amplitudes  of  electric  field  E  in  both 
sidelobes  lower  (A)  and  upper  (B)  would  be 

EA=EB  =  EjD(y)  (15) 

where  D(y)  is  the  directivity  of  the  antenna  at  an 
offset  y.  In  case  that  the  reflection  point  is 
located  at  distance  dm  and  altitude  and  the 
reflection  coefficient  is  p.exp(j%)  then: 

E A,  =  EjW)  +  IX-7  +  1 80  -  20) 

EBm=EjD(fj+$EylD(Y+ 180-2*) 


Fig  2.  Geometry  of  the  Reflection  on  curved  earth 


Then  the  difference  of  theese  two  terms  is 
proportional  to  antenna  directivities  at  these  two 
angles  It  has  been  shown  [6]  that  modified 
monopulse  antennas  can  eliminate  this  error. 
Also  is  shown  that  the  elevation  error  can  be 
either  positive  or  negative.  Also  simple  Sum- 
Delta  monopulse  antennas  with  vertical  sidelobe 
cancellers  might  be  used.  If  multiple  reflection 
points  do  exist  then  the  vector  sum  must  be  taken 
and  arbitarity  in  phase  will  most  likely  reduce 
the  effect. 

This  effect  is  strongly  reduced  if  the  target  is 
height-monitored  by  two  antennas 
simultaneously  for  the  same  reason. 


The  fact  is  that  in  a  T.I.R.N  even  at  intense 
multipath  environments  there  is  a  little 
possibility  of  deep  target  fading  because  of  the 
simultaneous  use  of  more  than  one  receivers  in 
any  case. 

6.  Clutter  environments. 

It  is  generally  known  that  little  bistatic  clutter 
data  are  available.  The  fact  that  antennas  with 
wide  horizontal  beam  angles  are  taken  into 
cosideration  leads  to  the  fact  that  large  clutter 
signals  are  to  be  expirienced  during  T.I.R.N.’s 
operation. 

From  some  bistatic  measurements  though  [7]  ,  it 
has  been  stated  that  bistatic  to  monostatic  clutter 
is  about  -3  to  -20  dB  lower  than  monstatic[8], 
Dopller  spread  using  either  Rayleigh  or  Weibull 
distributions  is  not  expected  to  exceed  that  of  a 
monostatic  radar. 

Clutter  rejection  though  can  be  better  by 
combining  MTI  or  Dopller  shift  measurements 
taken  by  the  receivers  and  exclude  (filter) 
measures  that  seem  quasi-static  to  all  the 
receivers. 

Anyway,  since  no  operational  T.I.R.N.  is  in  use 
currently,  simulations  (based  on  bistatic  and 
monostatic  measurements  availlable)  have  shown 
that  for  Rayleigh  and  Weibull  clutter 
distributions  4-receiver  to  monostatic  clutter 
rejection  is  about  6  to  10  dB  better  than  a 
monostaticradar  with  the  same  antenna  pattern. 

Conclusions. 

The  above  results  although  mathematically 
modelled  using  simple  Electromagnetic  wave 
propagation  theory  with  the  appropriate 
aproximations,  simmulation  results  with  EREPS 
and  RPOT  shareware  software  of  NOSC 
(U.S.Navy)  for  standard  atmosphere  and  some 
waveguide  cases  have  been  run  and  comply  well 
with  the  theoreticall  results  presented  here. 

Multipath  for  monopulse  antennas  and  Clutter 
cases  still  require  some  research  in  order  to  be 
fully  covered,  mostly  because  they  are  signal- 
dependent  and  to  this  time  only  low  PRF  and 
CW  have  been  examined. 

Anyway  the  capability  of  6-D  detection  of  a 
T.I.R.N.,  although  is  effected  by  environmental 
parameters  that  effect  other  types  of  radar  still 
remains  better  that  a  monostatic  radar.  Although 
computational  and  design  complexity  is  higher 


the  much  promising  performance  of  a  T.I.R.N. 
and  the  fact  that  it  is  totally  passive  makes  it  an 
attractive  system  model  for  further  development. 
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1.0  SUMMARY 

Modelling  results  from  a  programme  to  improve  the  accuracy 
of  the  prediction  of  ionospheric  parameters  are  presented. 
Ionospheric  conditions  impact  upon  communications, 
navigation  /  GPS,  satellite  operations  and  surveillance  radar 
operations.  As  a  result,  ionospheric  predictions  have  an 
important  bearing  upon  the  specification  of  the  battlespace 
environment. 

The  paper  presents  a  novel  and  robust  technique  that  can  cope 
with  the  problems  of  noise  and  non-contiguity  that  are 
endemic  to  solar-terrestrial  data  sets. 

The  method  of  using  non-linear  radial  basis  function  (RBF) 
neural  networks  (NNs)  to  model  the  noon-day  variation  of  the 
critical  frequency  of  the  F2  layer  of  the  ionosphere,  foF2,  is 
investigated.  A  technique  based  upon  singular  value 
decomposition  is  also  adopted  for  the  purposes  of  noise 
reduction.  The  performance  of  the  model  is  compared  with  the 
results  obtained  from  the  reference  persistence  model 
predictions.  Consecutive  noon-day  foF2  values  from  1957  to 
1990  from  the  Slough  ionosonde  station  (UK)  are  used  to  train 
and  test  the  model.  Predictions  are  made  for  timescales  of  one 
to  thirty  days  ahead,  using  both  the  RBF  and  persistence 
models.  Relative  performance  is  quantified  using  root  mean 
square  error  (RMSE)  between  the  RBF  and  persistence 
prediction  time  series  compared  with  the  actual  time  series, 
over  the  testing  interval. 

It  is  found  that  RBF  NNs  offer  a  significant  improvement, 
approximately  60%,  over  the  performance  of  the  reference 
persistence  model. 

2.0  INTRODUCTION 


Even  well  designed  systems  are  vulnerable.  Predictions  of  the 
ionised  battlespace  environment  are,  therefore,  of  paramount 
operational  significance.  The  effects  of  the  ionosphere  on  a 
number  of  military  systems  are  shown  in  Table  1 . 


TablelT^eeffectsoftheiono^hereonCS^s^steim 


i  Communications 

Surveillance 

Navigation 

Systems  •  HF  communications 
i  and  broadcasting 

UHF/SHF  radars 

GPS 

j  UHF/SHF  satellites 

HF  OTHR  radar 

LOREN/DECCA 

•  LEO  and  MEO  cellular  & 
i  dataSATCOM 

Spaced  based  SAR 

:  VLF-MF  communication 
!  and  broadcasting 

Geo-location 

Ef.fSJC.tS  :  Data/symbol  errors 

Range  and  bearing 
errors 

Range  errors 

Loss  of  phase 
lock  and  data 

:  Sky-wave/ground-wave 

I  interference 

Loss  of  target 
discrimination 

Sky-wave 

contamination 

1  Loss  of  communications 

Spectral  distortion 

Prohibits  remote 
sensing 

Loss  of  SAR  aperture 
phase  coherence 

•Severity  [  Can  lead  to  complete  HF 
i  blackout 

Many  km  of  range 
uncertainty  at  HF 

GPS  single  freq 
position  errors 
up  to  75  metres 

GPS  loss  of 
position  update 

•  Up  to  30Hz  Doppler  shift 

Bearing  errors  up  to 

100  degrees 

|  HF  multi-path  up  to  8ms 

Over  200  m  at  UHF 

j  30  dB  fades  at  400  MHz 

|  20  dB  fades  at  L-band 

Causes  •  Ionospheric  irregularities 

:  Multi-path 

:  Attenuation 

:  Doppler 

Total  electron  content 
variations 

Ionospheric 

irregularities 

Total  electron 

content 

variations 

Ionospheric 

irregularities 

2.1  Military  relevance 

2.2  Ionospheric  prediction 

Ionospheric  disturbances  can  effect  both  ground  and  space 

based  communications,  navigation  and  surveillance  systems.  a  long  standing  goal  of  solar-terrestrial  physics  has  been  the 
Examples  include  military  communications,  GPS,  counter  prediction  of  specific  events,  or  indices  of  activity,  that  impact 

stealth  HF  over-the-horizon  radar  and  ballistic  missile  defence.  upon  communications,  satellite  and  utility  operations. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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Correlation  studies  that  relate  solar  wind  energy  input  to 
various  indices  of  geomagnetic  activity  have  played  a  crucial 
role  in  understanding  the  sun  -  solar  wind  -  magnetosphere  - 
ionosphere  coupled  system.  In  addition,  linear  predictive 
techniques  have  been  used  to  deduce  relationships  between 
input  and  output  data  [Milsom  et  al.,  1987,  1989],  Although 
linear  techniques  have  achieved  some  measure  of  success,  the 
limit  of  these  approaches  appears  to  have  been  reached.  A 
number  of  studies  have  demonstrated  the  importance  of  non¬ 
linear  behaviour  within  the  sun- solar  wind-magnetosphere- 
ionosphere  system  [Baker  et  al.,  1990],  These  have  led  to  the 
development  of  non-linear  theoretical  models  [Klimas  et  al. , 
1992]  and  to  the  use  of  non-linear  time  series  methods 
[Vassiliadis  et  al.,  1992]  to  characterise  the  behaviour  of  these 
complex  systems. 

It  is  unlikely  that  linear  models  will  ever  adequately  describe 
the  complex  non-linear  magneto-hydrodynamic  (MHD) 
systems  that  govern  the  behaviour  of  the  solar-terrestrial 
environment.  These  coupled  MHD  equations  are  often 
insoluble  using  analytical  techniques.  In  the  absence  of  a 
tractable  solution  derived  from  the  full  MHD  equations,  it  is 
an  attractive  option  to  look  for  non-linear  features  in 
experimental  data  in  order  to  develop  models  using  empirical 
methods.  These  methods,  which  have  evolved  from  work 
begun  a  decade  ago  to  find  evidence  of  finite  dimensional 
dynamics  in  fluid  turbulence,  have  received  recent  attention. 
However,  they  have  been  developed  for  well-controlled 
experimental  systems  and,  therefore,  work  best  on  large  sets  of 
clean,  stationary  data. 

In  practice,  solar-terrestrial  data  sets  are  far  from  this  ideal; 
generally,  the  signal  to  noise  ratio  is  poor  and  the  data  is 
highly  quantised,  with  large  numbers  of  data  drop-outs  from 
the  time  series.  The  large  range  of  physically  significant  time- 
scales  also  means  that  the  time  series  are  rarely  stationary. 
Thus,  it  follows  that  naive  application  of  these  non-linear 
techniques  is  unlikely  to  succeed.  In  this  study,  we  attempt  to 
apply  novel  techniques  derived  from  studies  into  artificial 
intelligence  to  derive  more  robust  non-linear  predictive 
models. 

The  current  state  of  the  art  for  geophysical  forecasting 
[Joselyn,  1995]  falls  far  short  of  the  predictive  precision  that 
would  be  required  for  an  acceptable  model  of  future  solar- 
terrestrial  activity.  Any  prospective  model  must  be  compared 
with  the  reference  persistence  model  to  assess  its  relative 
performance.  The  latter  predicts  that  the  value  of  an 
observable  parameter  at  some  specified  point  in  the  future  will 
be  identical  to  the  currently  observed  measurement  of  that 
parameter.  This  model  performs  well  for  quiet  time  conditions, 
when  the  terrestrial  environment  is  relatively  undisturbed. 
However,  it  cannot  predict  the  onset  of  the  short-lived 
impulsive  disturbances  that  characterise  periods  of  unusually 
high  geomagnetic  and  ionospheric  activity.  A  successful 
predictive  model  must  offer  a  tangible  increase  in  performance 
over  the  persistence  technique  to  prove  the  value  of  the  method 
employed. 

Current  theoretical  and  empirical  models  fail  this  stringent 
criterion.  In  particular,  prediction  of  storm  events  and  spurious 
disturbances  are  very  poor.  Even  the  overall  performance  can 
be  inferior  to  the  standard  persistence  model.  This  shortfall  in 
the  performance  of  existing  models  can  be  attributed  to  several 


causes.  Firstly,  there  is  an  incomplete  understanding  of  the 
propagation  of  disturbances  from  their  solar  origins  to  the 
near-earth  environment.  Secondly,  the  coupling  between  the 
magnetosphere,  ionosphere  and  solar  is  not  well  understood. 
Finally,  very  few  continuous  time  series  exist  that  are 
indicative  of  solar  wind  activity  and  associated  solar  activity. 
The  combination  of  these  factors  makes  the  adoption  of 
knowledge  independent  modelling  techniques  highly  desirable. 

2.3  foF2  time  series 

The  critical  frequency  of  the  F2  layer  of  the  ionosphere  (see 
Figure  1),  foF2,  is  one  of  the  most  significant  parameters  of 
the  ionised  upper  atmosphere  for  the  purposes  of  military  and 
civil  applications.  As  this  area  often  contains  the  highest 
electron  concentration  within  the  ionosphere,  it  generally 
determines  the  maximum  usable  frequency  for  HF 
communications.  It  also  represents  the  usable  portion  of  the 
ionosphere  that  is  farthest  from  the  ground,  a  fact  that  has 
direct  bearing  on  the  maximum  usable  range  of  a 
communication  link.  For  a  fixed  elevation  angle,  transmission 
range  increases  as  the  maximum  height  of  the  point  of 
reflection  increases. 


Figure  1  Effects  of  the  ionosphere  upon  communications. 


The  foF2  time  series  is  determined  from  ionograms  (Figure  2), 
produced  by  ionosondes.  These  ionosondes  sweep  across  the 
HF  frequency  spectrum  and  produce  profiles  of  frequency  as  a 
function  of  virtual  height,  for  the  HF  frequency  band  (3  to  30 
MHz).  The  value  of  foF2  is  the  maximum  frequency  of  the 
ordinary  wave  that  is  returned  from  the  F2  layer  of  the 
ionosphere,  at  vertical  incidence.  A  global  network  of  such 
sounding  stations  record  measurements  of  foF2  every  fifteen 
minutes  and  compile  these  measurements  to  provide  hourly 
and  daily  averages,  in  conjunction  with  monthly  medians. 
These  values  are  available  from  a  number  of  World  Data 
Centres  (WDCs)  that  act  as  repositories  for  solar-terrestrial 
data.  World  maps,  such  as  the  example  given  in  Figure  3,  are 
then  constructed  to  provide  global  contour  plots  of  foF2  for  the 
HF  community.  The  WDCs  also  provide  on-line  predictions  of 
geophysical  activity  as  an  additional  service. 


The  present  study  utilises  the  foF2  time  series  corresponding 
to  the  ionosonde  station  at  Slough,  England.  Daily  noon  values 
of  foF2  for  the  period  1957  to  1990  were  used  to  train  the 
neural  network  model  and  test  its  effectiveness. 
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Figure  2  Vertical  Ionogram. 


Figure  3  Global  foF2  contour  map. 


The  time  series  as  a  whole  contained  approximately  10% 
missing  data  points.  These  missing  points  are  not  randomly 
distributed  through  the  time  series,  introducing  a  potential  bias 
into  the  model.  This  poses  a  significant  obstacle  for 
prospective  non-linear  prediction  schemes.  Therefore,  a 
procedure  has  to  be  defined  for  the  processing  of  input  training 
vectors  that  contain  a  missing  data  point.  This  scheme  should 
minimise  the  detrimental  effects  upon  the  model  of  including 
training  vectors  that  contain  a  missing  point. 

A  simple,  but  not  particularly  effective,  solution  is  to  set  all 
missing  data  points  to  a  value  of  zero,  subsequent  to  the 
normalisation  process.  This  normalisation  process  is  described 
in  the  discussion  of  the  analysis  techniques  employed  in  this 
study.  This  minimises  the  average  mean  square  interpolation 
error,  for  a  constant  value  interpolation,  over  the  de-meaned 
time  series  as  a  whole. 

The  next  level  of  sophistication  involves  the  use  of  adaptive 
linear  interpolation  techniques  that  examine  the  local 
behaviour  of  the  time  series  in  the  region  of  the  missing  point. 
Such  techniques  use  this  information  to  construct  a  linearly 
optimal  estimate  of  the  value  of  the  missing  data  point. 
However,  use  of  a  linear  interpolation  scheme  is  highly 
undesirable  for  time  series  that  display  significant  non-linear 
behaviour,  due  to  the  fragility  of  the  non-linear  properties  that 
have  to  be  characterised. 

This  study  has  chosen  to  adopt  non-linear  modelling 
techniques  with  the  aim  of  improving  the  prediction  of 


ionospheric  indices.  In  the  absence  of  a  suitably  robust  non¬ 
linear  method  of  dealing  with  missing  data  points,  it  is 
necessary  to  discard  those  training  vectors  that  contain  missing 
data  points. 

3.0  MATHEMATICAL  METHODS 

3. 1  Non-linear  Analysis 

Linear  techniques  have  proved  to  be  powerful  tools  in  the 
characterisation  and  prediction  of  time  series.  However,  there 
are  situations  in  which  these  techniques  are  unsuitable  and 
non-linear  techniques  need  to  be  considered. 

The  fundamental  principle  behind  non-linear  prediction 
techniques  is  Takens  theorem  [Tokens,  1981].  A  corollary  of 
which  is  that,  given  a  dynamical  system,  possibly  of  high  or 
infinite  dimension  that  has  a  stable  attracting  set  with 
dimension  d,  there  generally  exists  a  function,  requiring  at 
most  2d+\  independent  inputs,  that  can  be  used  as  a  predictor 
for  a  time  series  arising  from  that  system.  Takens’  result 
guarantees  the  existence  of  non-linear  predictive  functional 
models  of  the  data,  but  provides  no  details  about  the  dimension 
of  the  attractor,  or  how  to  calculate  the  underlying  function. 

3.2  Singular  value  decomposition 

Solar-terrestrial  data  sets  are  typically  noisy  and  singular  value 
decomposition  (SVD)  can  be  employed  to  pre-process  the  time 
series  to  allow  subsequent  selection  of  principal  components  to 
optimise  separation  of  the  signal  and  noise  subspaces. 
Removing  the  noise  subspace  improves  the  predictive 
performance  of  RBF  NNs,  as  described  in  section  3.3. 

SVD  is  a  powerful  technique  associated  with  matrix  based 
computations  and  analysis.  SVD  is  used  principally  to  provide 
an  optimal  solution  for  a  linear  least  squares  problem.  In 
addition,  SVD  can  also  be  used  to  devise  filters  for  noisy 
signals.  In  this  study,  the  matrices  used  in  the  calculations 
have  been  derived  from  the  time  series,  using  a  moving 
window  technique. 

For  an  ideal  error  free  measurement  system  (the  data  lies  in  a 
finite  dimensional  space),  measurement  data  can  be  arranged 
in  a  matrix,  where  the  matrix  is  known  to  be  rank  deficient. 
This  means  that  some  or  all  of  the  rows  or  columns  of  a  matrix 
are  linearly  dependent  upon  each  other.  This  linear 
dependency  breaks  down  when  any  form  of  noise  is  present,  as 
in  almost  all  actual  measurement  systems.  The  resultant 
measurement  matrix  often  becomes  full  rank  in  such  cases. 
SVD  can  be  used  in  these  instances  to  determine  a  set  of 
orthogonal  bases  that  minimise  the  effects  of  this  loss  of  linear 
dependency  between  the  rows  or  columns  of  the  matrix,  due  to 
noise.  In  essence,  SVD  provides  the  optimal  linear  separation 
of  the  signal  and  noise  subspaces  and  can  be  used  to  minimise 
redundancy  in  the  input  vector. 

To  achieve  this  optimal  estimate,  a  sliding  window,  of  length 
n,  is  passed  along  the  data  set,  one  point  at  a  time,  to  construct 
the  matrix  of  delay  vectors.  The  SVD  of  this  matrix  can  be 
used  to  create  a  set  of  n  orthogonal  filters  whose  summed 
output  would  produce  the  original  time  series.  Each  filter 
corresponds  to  a  principal  component  (PC).  The  magnitudes  of 
these  PCs  can  be  plotted  in  order  of  decreasing  size  to  assess 
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the  relative  linear  importance  of  the  output  of  each  of  these 
filters  in  characterising  the  variation  of  the  time  series.  The 
efficiency  of  the  predictive  scheme  can  then  be  optimised  by 
only  including  those  PCs  that  have  a  positive  effect  upon  the 
model  output  error.  Those  PCs  that  adversely  affect  the  model 
accuracy  are  removed. 

In  summary,  SVD  provides  a  useful  technique  for  filtering  a 
noisy  signal  and  provides  an  estimate  of  the  number  of  inputs 
required  to  characterise  the  underlying  attractor  of  a  dynamical 
system.  It  is  also  the  method  of  choice  for  solving  linear  least 
squares  problems,  a  property  we  will  exploit  in  the  next 
section. 

3.3  Radial  basis  function  neural  networks  (RBF  NNs) 

Neural  networks  address  the  problem  of  fitting  points  (X.,Y.), 
where  i=l..N,  to  a  model  of  the  form  Yn=f  (Xn).  Clearly,  for  a 
finite  data  set,  there  will  be  a  whole  class  of  functions,  F,  such 
that  any  f  eF  satisfies  these  relationships.  Matters  are  further 
complicated  by  presence  of  noise.  By  fitting  the  data  exactly, 
we  inadvertently  fit  this  noise  as  well.  As  a  result,  the  model  is 
likely  to  perform  sub-optimally  on  unseen  data,  as  the  unseen 
noise  will  not  be  predictable  from  that  contained  in  the  original 
data  set.  Any  process  that  minimises  the  contribution  of  noise 
to  the  model  will  necessarily  improve  the  predictive  accuracy. 

To  address  both  of  these  problems  simultaneously,  it  is  usual 
to  select  the  /  that  minimises  the  modelling  error  in  a 
restricted  class  of  functions.  Increasing  the  size  of  this  class 
will  result  in  a  decrease  in  the  error  calculated  over  the 
original  training  data.  However,  on  previously  unseen  test  data 
drawn  from  the  same  distribution  the  error  will  eventually 
begin  to  grow,  as  the  noise  in  the  original  data  is  modelled 
more  and  more  accurately.  At  this  point  the  approximation  is 
deemed  optimal,  since  it  models  the  general  trends  within  the 
data  without  modelling  the  noise. 

There  are  numerous  linear  and  non-linear  methods  for 
determining  such  functional  relationships  between  time  series. 
The  RBF  neural  network  offers  one  approach  to  the  solution  of 

this  problem.  Given  two  sets  of  data  points  Kt,  and 

{A}.  _j ,  it  is  desirable  to  construct  a  function  f(Xn)  that 
minimises  the  summation:- 

E  =  X  (  f(X„)  -  Yn)2  (1). 

n=l 

The  RBF  approach  reduces  the  construction  of  f(Xn)  to  that  of 
a  linear  least  squares  problem.  This  can  be  solved  using  SVD, 
by  assuming  f(XD)  to  be  the  linear  weighted  sum  of  radially 
symmetric  functions  of  X,  which  contain  the  non-linear 
adaptability  of  the  network,  that  is, 

f(Xn)  =  X^(|Xn-Ci|)  (2). 

i=l 

A  number  of  individual  basis  functions,  (p.  where  i=l,..,p  are 
chosen  with  centres  c  and  weights  <o.  to  construct  the  function 
f.  Given  this  formulation  of  the  problem,  equation  (1)  reduces 


to  a  linear  least  squares  problem  that  can  be  solved  by 
applying  SVD. 

In  the  results  presented,  cp  is  a  cubic  function  of  the  form, 

(p{r)  =  r3  (3). 

Applying  equation  (3)  reduces  equation  (1)  to  a  linear  least 

squares  problem  of  the  form  MW=Y,  where  W=[Wj, . ,w  ]  , 

Y=[Y,, . ,YJ  and. 


1 

~<3 

X 

1 

o 

••  Hxi  ■  cp 
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i 

Jp 

••  ?\x i  -  cp|_ 

(11). 

We  require  the  weights  w.,  so  solving  for  W  yields  W=NY, 
where  N=  (MtM)  ‘Mt  (if  p<n)  or  N=  MT(M  MT)‘'  (if  p  >  n), 
the  Moore-Penrose  inverse  of  a  non-square  matrix,  and  a  ‘-1’ 
indicates  the  matrix  inversion  operator. 

This  method  has  several  advantages  over  the  more  commonly 
used  Multi-Layer  Perceptron  (MLP)  techniques.  MLPs  train 
through  the  use  of  gradient  descent  algorithms,  which 
determine  optima  on  a  non-linear  surface  given  a  set  of  initial 
conditions.  There  is  no  guarantee  that  a  particular  optimum  is 
global  and  various  initial  conditions  must  be  tested  to  find  the 
best  optimum  solution.  Gradient  descent  also  relies  on  an 
iterative  process,  the  back-propagation  of  training  errors 
through  the  network  to  adjust  the  network  parameters  to 
provide  a  better  fit  to  the  training  data.  Cross  validation 
against  a  control  data  set  determines  when  the  network 
generalisation  capabilities  are  optimum,  for  a  certain  set  of 
initial  conditions.  Iterative  processes  are  typically  numerically 
intensive  and  arbitrary  to  a  certain  degree. 

RBF  NNs  are  assured  of  finding  the  global  optimum  solution 
in  a  single  pass,  because  the  training  process  is  a  linear  least 
squares  problem.  Thus,  RBF  NNs  are  relatively  cheap  in 
computational  terms  and  more  straightforward  to  use  than 
MLPs.  The  disadvantage  of  the  RBF  approach  is  that,  because 
its  form  restricts  the  search  to  a  linear  hyper-surface  in  the 
function  space,  it  can  require  more  degrees  of  freedom 
(centres)  to  perform  as  well  as  the  MLP.  This  can  sometimes 
be  an  issue  when  the  amount  of  data  is  limited.  In  general, 
however,  the  ease  of  fitting  the  RBF  more  than  compensates 
for  any  necessary  increase  in  the  complexity  of  the  class  of 
models.  In  addition,  RBF  NNs  provide  a  straightforward 
functional  relationship  between  input  and  output  vectors, 
which  facilitates  understanding  of  the  model  constructed  by 
the  network.  Rule  extraction  for  MLPs  is  a  much  more 
complicated  matter  and  the  information  contained  in  the 
knowledge  matrix  is  not  amenable  to  intuitive  decomposition. 

4.0  ANALYSIS  TECHNIQUE 

The  RBF  model  was  trained  and  tested  using  noon  day  foF2 
values  from  the  Slough  ionosonde  covering  the  period  from  01 
January  1957  to  31  December  1990.  The  total  length  of  the 
time  series  is  12,418  points.  The  first  10,000  were  used  to  train 
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the  RBF  model  while  the  remainder  of  the  points  were  used  to 
test  the  predictive  performance  of  the  model  on  unseen  data. 

The  time  series  as  a  whole  was  reduced  to  zero  mean  and  unit 
variance.  Normalisation  of  the  input  data  in  this  fashion 
removes  the  need  for  an  amplitude  or  DC  component 
parameter  input  for  the  radial  basis  functions  that  we  have 
adopted  for  the  purposes  of  this  study. 

A  sliding  window  of  length  90  days  was  passed  along  the  time 
series,  one  point  at  a  time,  to  construct  the  matrix  of  input 
vectors  that  are  required  to  train  the  RBF  model.  This  value 
was  selected  by  assessing  the  relative  performance  of  the  one 
day  ahead  prediction  model  using  a  range  of  input  window 
lengths.  Input  vectors  that  contain  one  or  more  missing  data 
points  are  then  removed  from  the  training  set.  For  this  study, 
10%  missing  data  points  gave  rise  to  approximately  90% 
rejection  of  possible  training  vectors.  The  training  vector  set 
was  then  projected  onto  its  principal  axes  using  SVD. 
Principal  components  that  adversely  affected  the  prediction 
were  removed. 

The  remaining  filter  outputs  were  then  used  to  train  the  RBF 
model.  The  optimum  number  of  centres  (functions)  was 
determined  by  assessing  the  relative  performance  of  the  one 
day  ahead  prediction  model  using  a  range  of  numbers  of 
centres.  A  value  of  140  centres  was  found  to  give  the  best 
results.  Models  were  then  constructed,  using  these  parameters, 
to  predict  from  1  to  30  days  ahead.  The  results  were  compared 
with  the  corresponding  persistence  model  prediction,  using 
root  mean  square  error  (RMSE). 

5.0  DISCUSSION  OF  RESULTS 

The  results  displayed  in  Figure  4  represent  the  relative 
performance  of  the  RBF  NN  model  in  terms  of  RMSE,  in 
comparison  with  the  reference  persistence  model,  for 
prediction  timescales  of  one  to  thirty  days  ahead.  For  a  one 
day  ahead  prediction,  the  RBF  model  (no  noise  filtering) 
produces  an  RMS  error  of  0.655  MHz,  in  comparison  with  an 
RMS  error  of  1.186  MHz  for  the  standard  persistence  model. 
This  represents  a  relative  increase  in  performance  of 
approximately  45%  over  the  persistence  model.  Optimising  the 
noise  filtering  process  led  to  the  removal  of  36  of  the  PCs  that 
adversely  affected  the  accuracy  of  the  model.  For  a  one  day 
ahead  prediction,  the  RBF  model  (with  SVD  noise  filtering) 
produces  an  RMS  error  of  0.484  MHz,  a  relative  increase  in 
performance  of  59%  over  the  standard  persistence  model. 

The  graph  shows  that  the  RBF  model  represents  a  significant 
improvement  over  the  standard  persistence  model.  This  is 
particularly  true  with  regard  to  predictions  up  to  ten  days 
ahead.  In  this  region,  the  predictive  accuracy  stays  roughly 
constant.  The  performance  increase  margin  becomes  narrower 
as  the  prediction  time-scale  increases  to  thirty  days  and  the 
model  error  rises,  but  noise  reduction  mitigates  this  problem. 
The  first  effect  can  most  likely  be  attributed  to  the  fact  that 
optimised  model  parameters  were  only  derived  for  the  one  day 
ahead  prediction.  This  was  necessary  due  to  limitations  on 
available  processing  resources.  These  optimal  values  will 
become  increasingly  inaccurate  as  the  prediction  extends 
further  into  the  future.  Better  results  would  be  obtained  if  the 
model  was  optimised  separately  for  each  of  the  number  of 
steps  ahead  required  for  the  prediction. 


Noise  reduction  provides  further  worthwhile  improvements, 
particularly  in  regard  to  the  long  range  and  one  step  ahead 
predictions.  The  noise  reduction  process  was  also  optimised 
with  respect  to  the  one  day  ahead  prediction,  hence  the  radical 
improvement  over  the  unfiltered  model  for  the  one  day  ahead 
model.  Much  smaller  improvements  are  observed  for  short 
time-scale  predictions  (2-10  days).  Apparently,  the  noise 
reduction  scheme  produced  for  the  one  day  ahead  prediction 
seems  to  be  too  specific  to  have  a  pronounced  effect  on  the 
other  short  term  models.  This  indicates  that  optimising  each 
model  separately,  with  regard  to  noise  reduction,  would  also 
provide  larger  improvements  in  terms  of  predictive  accuracy. 
However,  the  short  term  prediction  accuracy  plateau  now 
extends  to  15  days,  five  days  longer  than  for  the  corresponding 
unfiltered  RBF  model.  Noise  reduction  provides  tangible 
improvements  for  medium  time-scale  predictions  (10-25  days), 
giving  a  fairly  constant  improvement  in  accuracy  across  the 
whole  period.  The  effects  of  the  noise  reduction  scheme  upon 
the  longer  term  predictions  (25-30  days)  are  more  pronounced. 
This  arises  from  the  fact  that  prediction  error,  due  to  the 
effects  of  noise,  increases  with  the  time-scale  of  the  prediction. 
Hence,  any  noise  reduction  process  will  have  a  more 
pronounced  effect  upon  longer  term  predictive  models. 

Figure  5  represents  the  relative  difference  between  the  RBF 
model  and  the  persistence  model  for  each  of  the  types  of 
prediction.  This  allows  assessment  of  the  similarities  between 
the  model  produced  by  the  RBF  and  the  reference  persistence 
model,  for  varying  prediction  timescales.  The  effects  of  noise 
reduction  on  these  features  can  also  be  determined. 

The  divergence  of  the  unfiltered  RBF  model  from  the 
persistence  model  increases  markedly  after  twelve  days.  This 
point  coincides  with  the  narrowing  of  the  performance 
increase  margin  noted  in  regard  to  Figure  4.  This  feature  can 
also  be  observed  for  the  filtered  RBF  model.  However,  in  the 
latter  instance,  the  relative  difference  undergoes  a  dramatic 
decrease  after  25  days.  This  effect  is  coincident  with  the 
improvement  in  predictive  accuracy  over  the  unfiltered  RBF 
model  noted  in  Figure  4.  In  this  graph,  the  filtered  RBF  model 
is  seen  to  be  superior  to  the  unfiltered  RBF  model  for 
prediction  timescales  greater  than  25  days.  These  two 
observations  lead  to  the  deduction  that  the  noise  reduction 
process  improves  the  ability  of  the  RBF  to  successfully  model 
structure  that  can  be  attributed  to  the  recurrence  period  of  the 
solar  rotation  (25-30  days). 

Comparison  between  the  testing  data  set  and  the  prediction 
output  time  series  reveals  those  areas  where  the  RBF 
prediction  is  least  affective.  The  predicted  onset  of  periods  of 
unusually  high  geomagnetic  activity  lags  behind  the  actual 
time  series.  In  addition,  the  model  underestimates  the  peak 
magnitude  of  these  events  by  an  approximate  factor  of  two. 
This  gives  rise  to  large  error  deviations  in  these  regions  that 
can  account  for  much  of  the  overall  RMS  error.  However,  the 
performance  of  the  RBF  models  is  still  superior  in  this  respect 
with  regard  to  the  reference  persistence  model. 

Three  factors  could  account  for  the  relatively  poor  behaviour 
of  the  model  during  periods  of  elevated  geophysical  activity. 
Firstly,  periods  of  unusually  high  ionospheric  activity  are  rare 
in  comparison  with  quiet  time  conditions.  As  a  result,  the 
training  process  is  biased  away  from  the  prediction  of  these 
infrequently  occurring  events,  in  favour  of  accurate  prediction 
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of  quiet  time  conditions.  We  plan  to  rectify  this  problem  by 
increasing  the  relative  importance  of  non-quiet  time 
conditions.  Secondly,  the  timescales  associated  with  the 
evolution  of  such  disturbances  are  very  much  less  than  the 
resolution  of  the  input  time  series.  Therefore,  the  input  time 
series  does  not  contain  sufficient  information  concerning  the 
dynamics  of  these  events  to  accurately  model  their  rapid 
development.  Use  of  additional  higher  resolution  inputs  may 
be  required  to  address  this  issue.  Finally,  the  current  RBF 
model  relies  solely  upon  a  single  input  time  series  to  produce  a 
prediction.  Certain  data  sets  are  capable  of  providing 
precursive  information  that  has  a  direct  bearing  upon  the 
prediction  of  foF2.  Including  these  time  series  as  additional 
inputs  to  the  model  would  provide  a  measure  of  forewarning 
that  could  improve  the  prediction  accuracy  for  periods  of  high 
ionospheric  activity. 


Figure  4  Prediction  error  versus  no.  of  days  ahead  for 
prediction  of  foF2  noon  day  values  (RMSE). 


Relative  difference  (Residue)  between  RBF  and  persistence  for 
models  1 980  -  1 990  foF2  noon  day  values 


Figure  5  Relative  difference  between  RBF  and  persistence 
models  for  prediction  of  foF2  noon  day  values  (RMSE). 

6.0  CONCLUSIONS 

This  study  has  presented  a  novel  robust  technique  that  can 
cope  with  the  problems  of  noise  and  non-contiguity  that  are 
typical  to  solar-terrestrial  time  series. 

The  method  of  using  non-linear  radial  basis  function  (RBF) 
neural  networks  (NNs)  to  model  the  noon-day  variation  of  the 
critical  frequency  of  the  F2  layer  of  the  ionosphere,  foF2,  was 


implemented.  The  benefits  of  SVD  noise  reduction  were 
assessed  and  were  found  to  offer  a  substantial  improvement. 

The  performance  of  the  model  was  compared  with  the 
reference  persistence  predictions.  The  unfiltered  RBF  model 
produces  an  RMS  error  of  0.655  MHz,  which  is  approximately 
45%  more  accurate  than  the  corresponding  persistence 
prediction,  with  an  RMS  error  of  1.186  MHz.  For  a  one  day 
ahead  prediction,  the  RBF  model  (with  noise  filtering) 
produces  an  RMS  error  of  0.484  MHz,  a  relative  increase  in 
performance  of  59%  over  the  standard  persistence  model.  This 
accuracy  is  maintained  for  a  ten  to  fifteen  day  period  before 
the  error  starts  to  rise  significantly.  Optimising  the  models  for 
each  individual  prediction  time-scale  will  bring  further 
improvements  to  the  accuracy  of  the  predictions. 

In  conclusion,  the  RBF  NN  model  presented  in  this  study  is  a 
significant  improvement  over  the  standard  persistence  model, 
which  acts  as  the  reference  for  the  assessment  of  any  proposed 
prediction  scheme. 
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DISCUSSOR’S  NAME:  P.  Hoogeboom 

COMMENT/QUESTION: 

The  prediction  result  for  one  day  ahead  is  much  better  than  for  more  days  ahead.  Also,  the 
results  seem  to  indicate  a  cycle  at  4,  9, 14  days  ahead  of  slightly  improved  results.  Is  there  any 
explanation  for  these  observations? 


AUTHOR/PRESENTER’S  REPLY: 

Due  to  limitations  on  available  processing  resources,  it  was  only  possible  to  optimise  the  model 
parameters  for  the  one  day  ahead  prediction.  Hence,  these  results  show  a  much  greater  relative 
improvement  over  the  persistence  model,  compared  with  the  longer  timescale  predictions.  The 
observed,  4,  9,  14  day  cycles  may  well  be  an  artifact  of  this  inability  to  optimise  each  individual 
model.  So,  we  are  not  prepared  to  comment  upon  these  features,  in  terms  of  what  physical 
processes  they  might  represent,  until  the  cause  of  uncertainty  can  be  eliminated. 
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Fusion  de  donnees  et  fiabilite  des  transmissions  HF 
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Bat.  214  -  91405  Orsay  cedex 
France 


1.  SUMMARY 

High  frequency  reliable  radiocommunications  in  decametric 
band  is  currently  based  on  procedure  such  as  ALE  (Automatic 
Link  Establishment),  RTCE  (Real  Time  Channel  Evaluation) ... 
which  purpose  is  to  test  channel  transmission  capacity  and 
interferences  level. 

Establishment  tests,  made  in  real  time,  give  interesting  results 
by  using  long  time  experienced  procedures  with  very  good 
instantaneaous  reliabilities.  Though,  procedure  can  be  improved 
with  short  time  prevision  of  transmission  channel  characteristics 
and  with  interferences  forecasting  which  are  numerous  in 
decametric  band. 

Ionospheric  forecasting  methods  allow  to  take  into  account 
ionospheric  transmission  characteristic.  It  is  supposed  that  a 
very  short  time  forecasting  method  (1  to  2  hours)  or  that  a  short 
time  method  (24  to  48  hours)  is  implemented.  Interference 
forecasting  is  of  prime  importance  in  prevision  of  good 
establishment  of  links. 

Electromagnetic  spectrum  observation  in  HF  band  shows  that : 

•  Correlation  from  a  day  to  another  of  interferences  levels  are 
often  strong  (figure  1). 

•  Spectral  correlations  are  good  until  distance  of  about 
400/500  km  (figure  2). 

•  Interferences  levels  are  dependent  of  observation  azimuth 
(figure  3). 

•  Spectral  occupancy  is  correlated  with  critical  frequency 
FoF2. 

Taking  into  account  these  characteristics  lead  to  consider  the 
establishment  of  interferences  level  forecasting  and  the  critical 
frequency  of  F2  layer. 

Jamming  level  characterization  is  introduced  by  considering  the 
notion  of  clear  frequency  for  a  specified  level.  Frequency  is 
supposed  to  be  clear  for  a  threshold  S  if  the  interferences  level 
remains  lower  to  this  threshold.  Clearness  probability  can  be 
determined  for  each  traffic  band  and  we  can  plot  surface  of 
probability  that  a  clear  frequency  one  day  remains  clear  the 
following  days.  These  surfaces  plotted  as  a  function  of  the 
threshold  and  the  forecast  horizon  are  shown  figure  5. 

Correlation  between  spectral  occupancy  and  FoF2  frequency  is 
also  modelised.  Comparison  between  FoF2  frequency 
determined  by  zenithal  sounding  with  measurement  of 
occupancy  -  m,F  -has  be  proved  very  satisfying  (figure  6). 

A  data  fusion  processing  lead  to  a  totaly  passive  method  of  real 
time  network  management. 

2.  GENERALITES 

L’etablissement  de  radio-communications  fiables  dans  la 
gamme  decametrique  est  actuellement  base  sur  des  procedures 
de  type  ALE  (Automatic  Link  Establisment),  RTCE  (Real  Time 
Channel  Evaluation)  qui  visent  a  tester  les  capacites  de 
transmission  du  canal  et  les  niveaux  d’ interferences.  La 


communication  etant  etablie,  on  utilise  une  procedure  AME 
(Automatic  Maintain  Establisment)  qui  permet  de  degager  la 
liaison  de  la  frequence  dont  la  clarte  se  deteriore  lors  la 
transmission  de  donnees.  Les  tests  d’etablissement  sont 
effectues  en  temps  reel  et  donnent  des  resultats  interessants  en 
utilisant  des  procedures  dont  Fanciennete  leur  confere 
d’excellentes  fiabilites  instantanees.  Ces  procedures  presentent 
cependant  l’inconvenient  de  necessiter  des  mesures  sur  le  canal 
et,  par  consequent,  Tutilisation  d’emission  d’ondes 
electromagnetiques.  Par  ailleurs,  la  capacite  de  transmission  du 
canal  n’est  pas  le  seul  parametre  a  prendre  en  compte  et 
l’occupation  spectrale,  particulierement  importante  dans  ce 
domaine  de  frequences,  impose  d’en  tenir  compte. 

On  peut  observer  que  le  spectre  des  ondes  decametriques 
s’etendant  de  3  a  30MHz  est  particulierement  encombre.  On 
denombre  plusieurs  centaines  de  milliers  d’emissions 
radioelectriques  simultanees  sur  le  globe.  Ces  emissions 
constituent  une  banque  de  donnees  particulierement  riche  que 
l’on  peut  exploiter  de  deux  faqons  differentes  : 

Tout  d’abord,  on  doit  remarquer  que  ces  emetteurs 
constituent  des  signaux  parasites  qui  genent  les 
communications. 

On  doit  done,  pour  Fetablissement  des  communications, 
rechercher  parmi  les  frequences  attributes  a  un  utilisateur, 
celles  qui  seront  susceptibles  d’etre  peu  brouillees.  Cette 
action  consiste  a  rechercher  les  frequences  claires  -  e’est- 
4-dire  celles  pour  lesquelles  le  niveau  des  interferences  est 
inferieur  a  un  seuil  donne  -  mais  il  convient  de  connaitre 
quelles  sont  les  periodes  de  temps  pendant  lesquelles  cette 
frequence  demeurera  claire.  Cette  approche  ne  peut  qu’etre 
probabiliste  et  il  est  done  usuel  de  determiner  les 
probability  de  clarte  de  chacune  des  frequences  se 
trouvant  dans  le  spectre. 

L’ensemble  de  ces  frequences  peut  etre  utilise  comme  une 
banque  de  donnees  pour  determiner  les  caracteristiques 
generates  du  milieu  de  propagation.  Cette  approche  peut 
etre  justifiee  par  le  fait  que  les  utilisateurs  s’adaptent  aux 
conditions  ionospheriques  avec  de  plus  en  plus  de 
precision  et,  par  consequent,  suivent  revolution  du  milieu. 
On  a  done  songe  a  utiliser  les  caracteristiques 
d’encombrement  du  spectre  decametrique  pour  en  deduire 
un  parametre  capital  dans  les  telecommunications 
ionospheriques,  a  savoir  la  frequence  critique  FoF2  de  la 
region  F2  de  l’ionosphere. 

Cette  approche  permet  done  de  realiser,  d’une  part  une 
provision  de  l’etat  de  Tionosphere,  d’autre  part  une  prevision 
des  interferences  par  une  methode  totalement  passive.  Les 
resultats  obtenus  dans  cette  etude  montrent  que  les  interferences 
sont  corrdlees  dans  le  temps  et  dans  l’espace  et  que  le  spectre  de 
la  gamme  decametrique  est  correle  avec  l’etat  de  Tionosphere. 
Ces  resultats  permettent  de  realiser  une  application 
operationnelle. 

3.  CARACTERISATION  DES  INTERFERENCES 

Une  etude  sur  les  interferences  a  ete  entreprise  au  LETTI  des  le 
debut  des  annees  1980  afm  de  tenter  de  les  caracteriser.  Des 
mesures  permanentes  ont  ete  faites  et  ont  permis  de  constituer 
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une  banque  de  donnees  s’etalant  sur  plus  d’un  cycle  solairc  et 
comprenant  des  centaines  de  millions  de  mesures.  A  partir  de 
cette  banque  de  donnees,  une  statistique  a  etc  faite  pour 
determiner  les  principals  caracteristiques  des  interferences. 

On  s’est,  dans  un  premier  temps,  orienter  sur  l’etude  des 
interferences  en  Europe  occidcntale.  On  a,  pour  cela,  utilise 
plusieurs  stations  de  mesures  compatibles  pour  permettre  des 
comparaisons  entre  plusieurs  sites.  Le  premier  objectif 
poursuivi  a  ete  d'examiner  si  ces  correlations  existaient  dans  les 
parametres  caracterisant  des  interferences.  On  a  pour  cela  utilise 
plusieurs  types  de  mesures. 


ainsi  que  la  bonne  correlation  entre  la  station  STUDIO  1  et 
STUDI02  distantes  de  450km  revelee  par  ces  enregistrements. 


.  ,  STUDIO  2  STUDIO  1 

Level  (dBm)  Level  (dBm) 

Frequency  7.2000  MHz  ,  Frequency  7.2000  MHz 


V!V' 


Frequency  8.6575  MHz 


Frequency  8.6575  MHz 


3.1.  Correlation  temporelle  des  interferences  [1] 

On  a  examine  si  les  interferences  presentaient  des  coherences 
dans  le  temps  et  des  coherences  dans  l’espace.  La  figure  1 
represente  quatre  spectres  obtenus  en  deux  sites  differents 
distants  de  450km,  l’un  situe  dans  la  region  sud  de  Paris,  l’autre 
situe  dans  le  sud-ouest  de  la  France  (Les  Eyzies  de  Tayac). 
Chaque  spectre  represente  un  ensemble  de  15  spectres 
superposes  mesures  sur  une  duree  de  1  heure  dans  la  gamme  1  a 
30MHz.  Deux  spectres  ont  ete  releves  a  23h00,  deux  autres  a 
14h00  T.U.  On  peut  constater  que  les  15  spectres  superposes 
conservent  une  bonne  coherence  ce  qui  signifie  que  les 
caracteristiques  d’ensemble  du  spectre  dccametrique  sont 
relativement  bien  coherentes.  L’examen  des  spectres  montre 
egalement  Pemplacement  des  bandes  d’emission  dc  radio¬ 
diffusion.  On  peut  noter  que  de  jour  le  spectre  est  plus  etendu 
vers  les  hautes  frequences,  ce  qui  est  provoque  d’unc  part  le 
milieu  et  d’autre  part  par  l’adaptation  des  utilisateurs  aux 
evolutions  du  milieu.  On  peut  remarquer  egalement  une 
absorption  des  basses  frequences  plus  importante  en  milieu  de 
joumee  que  la  nuit.  Ces  enregistrements  confirment  les 
phenomenes  connus  d’absorption  et  de  variations  de  l’ionisation 
du  canal  ionospherique  mais  montrent  egalement  une  bonne 
correlation  temporelle  sur  Fechelle  d’une  heure  de  la  globalit6 
des  spectres  ainsi  qu’une  grande  coherence  spatiale  qui  est 
visible  par  la  comparaison  des  spectres  obtenus  aux  stations 
STUDIO  1  (Paris)  et  STUDI02  (sud-ouest  de  la  France). 
L’analyse  de  la  coherence  spatiale  a  ete  faite  sur  des  distances 
variant  de  0  a  1500km  et  l’on  a  pu  constater  que,  jusqu’a  des 
distances  de  l’ordre  de  500km,  on  peut  consid6rer  que  les 
spectres  sont  bien  correles,  comme  le  montre  le  resultat  de  la 
figure  1.  Une  analyse  plus  fine  de  la  coherence  spatio- 
temporelle  peut  etre  faite  en  observant  des  bandes  de  frequences 
etroites.  Ce  type  d’observation  a  ete  fait  sur  de  nombreuses 
emissions. 


Level  (dBm)  Level  (dBm) 


Figure  1  :  Correlation  spatiale  ente  spectres  HF  mseuree  en  des 
points  distants  de  450  km 

La  figure  2  montre  les  evolutions  sur  deux  jours  et  demi  de 
signaux  occupant  des  bandes  de  frequences  de  largeur  de  1kHz 
a  quatre  frequences  differentes.  On  peut  noter  l’existence  tres 
nette  d’une  coherence  temporelle  avec  un  cycle  de  24  heures 


Frequency  6.8825  MHz 


Frequency  6.8825  MHz 


Frequency  7.0500  MHz 


Frequency  7.0500  MHz 


Measurements  period  :  31/08/90  12h  U.T.  to  03/09/90  OOh  U.T. 


Figure  2  :  Correlation  temporelle  et  spatiale  de  spectres  HF 
entre  STUDIO  1  et  STUDI02 

3.2.  Variations  azimutales  [1] 

Une  etude  de  F  intensity  des  brouilleurs  a  ete  effectuee  en 
fonction  de  leur  angle  d’arrivee.  On  a  utilise  pour  cela  un  reseau 
d’antennes  compose  de  32  antennes  large  bande  couvrant  la 
gamme  dccametrique  et  dont  le  lobe  est  orientable  dans  toutes 
les  directions.  La  resolution  typique  de  ce  reseau  est  de  15 
degres  a  15MHz.  On  a  fait  une  etude  statistique  de  Fintensite 
des  champs  en  fonction  de  Tangle  azimutal  d’arrivee  des  rayons 
pour  toute  la  gamme  de  frequences  en  etablissant  la  statistique 
sur  des  bandes  de  1MHz  et  en  ramenant  Fintensite  du  champ  a 
une  bande  de  3kHz.  Les  rCsultats  obtenus  sont  illustres  par  les 
diagrammes  de  la  figure  3  qui  montrent  les  valeurs  obtenues 
dans  3  bandes  de  frequences  differentes:  10- 11  MHz,  14- 
15MHz,  16-17MHz,  pour  quatre  heures  de  la  joumee  :  14h00, 
19h00,  OlhOO  et  07h00.  On  peut,  en  premier  lieu,  noter  l’effet 
joumalier  des  variations  du  milieu  de  propagation 
particulierement  visible  par  le  fait  qu’4  OlhOO  T.U.  le  niveau 
des  champs  regus  est  quasiment  nul  pour  les  frequences 
comprises  entre  14  et  17MHz  et  extremement  faible  pour  les 
frequences  comprises  entre  10  et  11  MHz.  On  peut  constater 
egalement  l’effet  de  lever  et  coucher  de  soleil,  particulierement 
visible  sur  la  bande  16-17MHz.  Ces  releves  revelent  clairement 
que  Fintensite  des  interferences  en  provenance  de  la  direction 
Est  est  notablement  plus  importante  que  celle  qui  provient  de  la 
region  Ouest  ou  Nord-Ouest.  Cette  difference  s’explique  sans 
difficulty  en  observant  que  beaucoup  plus  d’emetteurs  se 
trouvent  evidemment  localises  a  l’Est  de  la  region  parisienne 
qui  correspond  a  une  zone  terrestre,  qu’a  l’Ouest  de  la  region 
parisienne  qui  correspond  a  une  zone  maritime. 
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Centre  des  cercles  :  -80dBm/3kHz,  Difference  entre  2  cercles  : 
lOdBm 

Figure  3  :  Distribution  azimutale  de  puissance  regue  dans  une 
bande  de  1  MHz  et  convertie  dans  me  bande  3  kHz 

3.3.  Analyse  des  resultats 

Les  resultats  obtenus  au  cours  de  cette  etude  ont  montre  que  la 
correlation  temporelle  des  interferences  etait  bonne  ce  qui 
traduit  le  fait  que  les  utilisateurs  des  transmissions  haute 
frequence  par  voie  ionospherique  reglent  leurs  echanges  de 
fai;on  tres  ordonnee  en  s’attribuant  en  particulier  des  heures  de 
vacations.  11  va  de  soi  que  ce  phenomcne  est  particulierement 
marque  dans  les  zones  de  radiodiffusion  mais  ce  phenomene  est 
egalement  observe  de  faqon  extremement  constante  pour  les 
liaisons  a  caractere  professionnel. 

La  cohdrence  spatiale  constatee  laisse  penser  que 
1’echantillonnage  spatial  des  mesures  d’ interferences  peut  etre 
effectue  avec  un  pas  relativement  large  de  l’ordre  de  500km 
pour  en  avoir  une  cartographic  complete  sur  une  zone  donnee. 

La  repartition  azimutale  des  angles  d’arrivee  des  interferences 
apporte  la  preuve  que  1 ’utilisation  d’aeriens  directifs,  voire 
adaptatifs,  est  particulierement  utile  dans  le  domaine 
decametrique. 

Les  spectres  presentes  concement  l’Europe  occidentale.  II 
semble  que  pour  cette  region  les  resultats  soient  relativement 
coherents.  II  est  certain  que  l’encombrement  spectral  varie  selon 
le  point  d’observation  que  1’on  se  fixe  sur  le  globe.  On  a 
effectue  des  mesures  dans  l’ocean  pacifique  et  on  a  constate  que 
l’encombrement  spectral  etait  beaucoup  plus  reduit.  Les 
resultats  de  l’etudc  presentee  dans  cette  communication  sont 
done  a  considerer  comme  valables  sur  la  zone  Europe 
occidentale  et  qu’il  conviendrait,  pour  une  autre  zone 
geographique,  de  s’appuyer  sur  une  banque  de  mesures 
differente. 

4.  MODELISATION  DES  INTERFERENCES 

4.1.  Moderation  des  interferences  [2] 

Pour  caracteriser  les  evolutions  temporelles  de  l’occupation 
spectrale,  nous  avons  6value  pour  une  frequence  d’observation 
donnee  la  probabilite  pour  qu’a  une  heure  donnee  le  niveau  de 
1’ interference  presente  ne  depasse  pas  un  seuil  S.  On  a  effectue 
ces  observations  sur  des  periodes  minimales  de  15  jours  et  sur 
des  periodes  maximales  de  deux  mois  et  demi.  Afin  d’effectuer 
une  etude  quantitative  on  a  effectue  ces  calculs  pour  les  95 
bandes  de  frequences  allouecs  pour  couvrir  la  bande 
decametrique  (bande  de  radio-diffusion,  liaisons  fixes,  liaisons 
mobiles,  liaisons  maritimes  ...).  Pour  chacune  des  bandes 


observees  on  a  constitue  des  tableaux  A(Nj,  N,,  Nj)  composes 
d’clements  a(j,  t,  J)  donnant  le  niveau  des  interferences  et  ou  : 

-  Nj  nombre  de  jours  d’observation  (Nj  =  15) 

-  Ni  nombre  d’echantillons  dans  une  heure  (N,  =  30) 

-  nombre  de  points  de  frequence  dans  la  bande  analysee. 

Pour  un  seuil  fixe  S  on  forme  un  second  tableau  B(Np  Nb  Nj) 
dont  les  elements  b(j,  t,  f)  prennent  les  valeurs : 

b(j ,t,j)=0  si  a(j,  t,J)>S 

b(j,  t,f)  =  \  si  a(j,  t,j)<S 

La  probabilite  pour  que  la  frequence  soit  claire  pendant  J  jours 
consecutifs  pour  le  seuil  S  est  alors  donnee  par  : 

Nj-J+ 1  N,  Nf  j+j-l 

EES  YlKutJ) 

7=1  M  /='  i=J _ 

(Nj-J  +  l)NtNf 

La  figure  4  montre  une  evolution  typique  de  la  probabilite  de 
clarte  P(S,  J)  en  fonction  de  Jet  pour  differentes  valeurs  de  S. 
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Figure  4  :  Probabilite  de  clarte  en  fonction  du  nombre  de  jours 
J 

On  peut  noter  que  la  valeur  correspondant  a  J  =  0  represente  la 
probabilite  de  non  clarte  ce  qui  correspond  au  complement  de  la 
probabilite  de  clarte  sur  un  jour.  On  observe  d’autre  part  que 
pour  les  valeurs  de  J  superieures  a  1  la  decroissance  de  la 
courbe  de  probabilite  est  reguliere,  monotone  et  d’allure 
exponentielle.  Cette  constatation  nous  a  conduit  a  approcher 
cette  courbe  par  une  loi  de  la  forme  : 

P(S,  J)  =  exp  (-  A  J- B) 

ouAetB  sont  des  fonctions  des  seuils  S.  Une  etude  statistique 
a  montre  que  cette  approximation  est  tout  a  fait  acceptable  et 
que  les  coefficients  A  etB  pouvaient  etre  exprimes  simplement 
en  fonction  du  seuil  S.  On  aboutit  ainsi  a  l’expression  de  la 
probability  de  clarte  donnee  par  la  relation  suivante  : 

P(S,  J )  =  exp[-  exp(-  AtS  -  A2)-J  -  exp(-  BtS  -  B2 )] 


P(S,J ) 
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Les  coefficients  A\,  A 2,  B\,  B2  permettent  done  de  decrire 
totalement  la  probabilite  P(S,  J)  et  une  etude  de  la  valeur  de 
ces  coefficients  a  ete  entreprise.  Ces  coefficients  varient  en 
fonction  de  la  bande  analysee  et  en  fonction  du  temps.  La 
probabilite  de  clarte  de  P(S,  J)  peut  etre  representee  par  une 
surface  en  fonction  de  J  et  S.  La  figure  5  montre  une 
representation  de  ces  surfaces  pour  la  bande  de  frequence 
12330kHz  a  12401kHz  pour  la  periode  du  25  fevrier  1993  au  8 
mars  1993. 


Figure  5  :  probabilite  de  clarte  en  fonction  du  temps  et  du  seuil 


On  peut  noter  sur  ces  figures  que  la  surface  comprend  trois 
zones  :  celle  correspondant  au  niveau  S  eleve  pour  laquelle  la 
probabilite  de  clarte  est  voisine  de  1,  celle  pour  laquelle  les 
seuils  sont  faibles  et  la  probabilite  de  clarte  est  voisine  de  ziro 
et  la  zone  intermediate  ou  Ton  peut  voir  que  les  variations  de  la 
probabilite  de  clarte  ont  bien  des  allures  exponentielles.  On  peut 
noter  qu’au  dela  d’une  douzaines  de  jours  les  courbes  semblent 
s’ecarter  d’allure  exponentielle.  Ce  phenomene  s’explique 
aisement  en  considerant  que  la  population  sur  laquelle  est  faite 
la  statistique  decroit  en  fonction  de  la  profondeur  de  prevision  et 
qu’au  dela  dc  12  a  15  jours,  le  nombre  de  frequences  claires 
devient  trop  faible  pour  que  la  statistique  soit  valable.  C’est  la 
raison  pour  laquelle  il  a  ete  jug£  normal  de  limiter  les 
profondeurs  de  prevision  a  des  valeurs  de  7  jours  qui 
correspondent  a  un  compromis  acceptable.  Pour  tester  la 
validite  de  cette  methode,  on  a  recherche  sur  les  courbes  reelles 
les  valeurs  des  coefficients  A\,  A2,  B\,  B2  etablies  sur  des 
valeurs  de  7  jours  et  pour  toutes  les  heures  de  la  joumee.  A 
partir  du  calcul  d’une  surface  de  regression  on  a  calcule  les 
coefficients  de  correlation  Ra  et  R/,  entre  les  valeurs  de  A  et  B 
obtenues  par  une  mesure  directe  de  chacune  des  courbes  de 
probabilite  de  clarte  et  avec  les  coefficients  deduits  de 
l'ensemble  de  la  surface. 

Le  tableau  1  montre  que  les  coefficients  de  correlation  qui  sont 
obtenus  sont  superieurs  a  0,96  et  souvent  voisins  de  0,98  ou 
0,99. 


T.U 

A1 

A2 

B1 

B2 

Ra 

Rb 

0 

0,106 

13,640 

0,092 

11,580 

-0,964 

-0,994 

1 

0,106 

13,710 

0,087 

11,080 

-0,959 

-0,995 

2 

0,104 

13,420 

0,091 

11,470 

-0,966 

-0,992 

3 

0,103 

13,340 

0,090 

11,320 

-0,967 

-0,989 

4 

0,104 

13,430 

0,085 

10,830 

-0,956 

-0,995 

5 

0,112 

14,230 

0,095 

11,860 

-0,976 

-0,997 

6 

0,108 

13,510 

0,113 

13,440 

-0,989 

-0,996 

7 

0,099 

12,320 

0,099 

11,790 

-0,991 

-0,998 

8 

0,097 

12,220 

0,101 

12,060 

-0,988 

-0,993 

9 

0,087 

11,070 

0,095 

11,410 

-0,992 

-0,995 

10 

0,099 

11,620 

-0,978 

-0,987 

11 

0,089 

10,490 

-0,989 

-0,996 

12 

0,095 

11,230 

-0,979 

-0,987 

13 

0,107 

12,330 

-0,957 

-0,963 

14 

0,099 

11,430 

-0,980 

-0,989 

15 

■ 

0,097 

11,640 

-0,988 

-0,984 

16 

0,097 

11,560 

-0,990 

-0,984 

17 

0,011 

12,990 

-0,988 

-0,996 

18 

jfnC 

0,132 

15,800 

-0,994 

-0,997 

19 

IB 

0,109 

13,860 

-0,969 

-0,982 

20 

0,117 

0,109 

13,250 

-0,980 

-0,996 

21 

0,121 

0,105 

12,750 

-0,989 

-0,999 

22 

II  | 

0,089 

11,190 

-0,977 

-0,996 

23 

m 

0,083 

10,550 

-0,956 

-0,999 

Tableau  1  :  Probabilite  de  clarte 

Ce  resultat  quantitatif  prouve  la  validite  de  la  methode  par 
Pexcellente  correlation  obtenue  entre  la  modelisation  et  les 
mesures. 

3.2.  Application  a  la  recherche  de  frequences  claires 

La  modelisation  des  interferences  obtenues  par  cette  methode  a 
conduit  a  la  determination  d’une  methode  de  prevision  a  court 
terme  des  interferences.  La  methode  consiste  a  mesurer 
l’encombrement  spectral  pendant  6  jours  consecutifs  puis  a 
utiliser  ces  informations  pour  estimer  l’encombrement  au  jour 
J.  Dans  ce  but,  on  a  defini  une  probabilite  de  clarte  C(S,  J) 
pour  qu’une  sequence  claire  6  jours  consecutifs  soit  egalement 
claire  le  J+lleme  jour.  Un  calcul  simple  permet  de  montrer  que 
C(S,  J)  s’exprime  par  la  relation  : 

C(S,  J)  =  exp(-  A)  =  exp[ -  exp( -AtS  -  A?J] 

II  apparait  ainsi  que  l’indice  de  clarte  est  independant  du 
nombre  de  jours  d’observation  et  ne  depend  que  du  seuil  S  pour 
une  frequence  donnee  et  une  heure  donnee.  Cette  methode, 
appliquee  aux  cas  de  prevision  reelle  a  donne  des  resultats  tout  a 
fait  satisfaisants  traduits  par  la  figure  6  qui  montre  la 
comparaison  entre  les  probabilites  mesurees  et  les  probabilites 
etablies  a  partir  des  previsions. 


C(SJ)  Experimental 


Figure  6  :  Courbes  theorique  et  calculee  de  l  ’indice  de  clarte 
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Les  resultats  obtenus  montrent  que  la  prevision  des 
interferences  a  court  terme  peut  etre  effectivement  envisagee 
avec  l’espoir  d’obtenir  des  resultats  satisfaisants.  La  methode 
apparait  particulierement  simple  a  mettre  en  oeuvre  puisqu’elle 
necessite  qu’une  station  experimentale  legere  constitute  d’un 
recepteur  pilote  par  un  calculateur  afin  d’etablir  les  statistiques 
et  associe  a  une  antenne  de  caracteristiques  connues.  Les 
previsions  peuvent  etre  effectutes  a  partir  d’une  periode 
d’observation  de  6  jours  qui  permet  de  faire  une  projection  sur 
le  jour  suivant.  Cette  prevision  a  24  heures  peut  etre  affinee  par 
une  prevision  plus  fine  qui  prend  en  compte  les  fluctuations  de 
la  joumee  consideree  par  rapport  aux  joumees  precedentes. 
Cette  methode  permet,  en  affmant  les  resultats,  d’obtenir  des 
previsions  d’une  meilleure  quaiite. 

5.  CARACTERISATION  DU  CANAL  IONOSPHERIQUE 

L’allure  generale  du  spectre  HF  varie  dans  le  temps  et,  comme 
le  montre  les  spectres  represents  sur  la  figure  1,  on  peut 
constater  que  la  correlation  spatiale  est  suffisamment  bonne.  On 
a  done  songe  a  utiliser  la  forme  de  ce  spectre  pour  realiser  une 
modelisation  de  l’ionosphcre.  11  est  evident  que  la  seule 
observation  du  spectre  ne  peut  permettre  de  definir  avec 
precision  le  profil  de  l’ionosphcre  sans  emettre  des  hypotheses 
supplementaires.  On  s’est  attache  a  examiner  si  l’allure  generale 
de  ce  spectre  etait  en  relation  avec  la  frequence  critique  de  la 
region  F2  qui  apparait  comme  le  parametre  principal  de 
1’  ionosphere.  Pour  proceder  a  cette  etude  on  a  volontairement 
separe  les  spectres  representant  les  zones  de  radio-diffusion  des 
autres  spectres.  En  effet,  les  emissions  dans  les  bandes  de  radio¬ 
diffusion  sont  de  puissances  importantes  et  ne  peuvent  etre 
traitees  avec  les  autres  spectres.  L’etudc  presentee  peut  etre 
appliquee  aux  seuls  spectres  de  radio-diffusion  qui  constituent 
un  echantillonnage  du  spectre  decametrique  ou  a  1’ ensemble  des 
autres  emetteurs.  Les  resultats  presents  concernent  ceux  etablis 
sur  l’ensemble  des  autres  dmetteurs.  II  est  possible  de  definir 
des  modeles  de  spectres  a  partir  de  considerations  theoriques. 
Ces  considerations  qui  ont  et  faites  ont  finalement  montr6 
qu’elles  s’accordaient  assez  pauvrement  avec  la  realite.  C’est  la 
raison  pour  laquelle  on  a  bati  une  methode  plus  pratique  qui 
s’avere  donner  de  tres  bons  resultats. 

Cette  methode  consiste  a  definir  une  densite  de  probabilite  par 
bandes  de  frequences  allouees.  Cette  densite  de  probabilite  est 
donnee  par  la  relation  : 

Pf(fi)  = 

dans  laquelle  : 

B(fi)  represente  la  largeur  de  bande  de  la  I'eme  bande 
allouee  centree  sur ft 

P(fi)  represente  la  puissance  totale  re9ue  dans  la 
bande  B(fi) 

On  definit  alors  le  moment  du  premier  ordre  de  cette  fonction 
densite  de  probabilite  a  l’aide  de  la  relation  : 

mxF  =  YJf,pfW 

l 

Ce  parametre  mtF  se  trouve  etroitement  correle  avec  la 
frequence  critique  FoF2  de  l’ionosphere. 

Ce  resultat  est  illustre  (figure  7)  par  deux  enregistrements 
effectues  par  le  sondeur  zenithal  de  la  station  fran?aise  de 
Poitiers  et  par  le  releve  des  spectres  decant  triques  effectue  a  la 
station  STUDI02  du  LETTI  situee  a  150  km  de  Poitiers.  Sur 
ces  enregistrements  on  a  fait  apparaTtre  les  variations  du 
parametre  mtF  deduit  des  spectres  decametriques.  On  peut 
noter  une  excellente  correlation  entre  ce  parametre  ttl\F  et  la 
frequence  critique  FoF2.  Cependant,  il  doit  etre  remarquer 


p(fmfi) 

Y^Pirmfi) 


egalement  que  lors  des  transitions  jour-nuit,  nuit-jour  les 
courbes  de  mtF  s’ecartent  sensiblement  de  la  courbe  EoF2.  Cet 
ecart  s’explique  par  le  fait  que  les  emetteurs  captes  proviennent 
pour  une  part  importante  de  la  direction  Est,  comme  le  montre 
la  figure  3,  et  par  consequent  les  utilisateurs  se  trouvent  «  en 
avance  »  par  rapport  au  point  d’observation  situe  en  France. 
Cette  difference  est  systematique  et  a  ete  done  traitee  comme  un 
biais.  Ce  biais  a  ete  estime  de  faqon  a  pouvoir  le  corriger 
systematiquement. 


0  I  ?  3  4  S  G  7  8  9  10  II  12  !3  M  IS  IG  17  16  19  78  21  22  23  24 


Figure  7  :  Correlation  entre  les  mesures  FoF2  et  F0F2  deduit 
du  spectre  HF 


Afin  de  pouvoir  evaluer  les  performances  de  cette  mesure  de 
FoF2,  on  a  introduit  deux  parametres  : 

L’un  representant  l’ecart  entre  la  frequence  critique 
FoF2  vraie  et  lc  parametre  m^F : 

X=  foF2vraie  -  m\F 

L’autre  representant  l’ecart  entre  la  frequence  critique 
FoF2  et  la  frequence  foF2  estimee  par  des  previsions  a 
90% : 

Y  =  FoF2vraie  -  foF2  a  90% 

Pour  l’ensemble  des  mesure  effectuees,  on  a  calcule  l’ecart 
moyen  et  l’ecart  quadratique  moyen  des  deux  quantites  X et  Y. 
Les  resultats  ont  donne  : 

Sans  correction  du  biais 


Ecart  moyen  de  X : 
Ecart  type  moyen  de  X : 
Ecart  moyen  de  Y : 

Ecart  type  moyen  de  Y : 
Avec  correction  du  biais 

Ecart  moyen  de  X : 
Ecart  type  moyen  de  X : 


/£ c  =  0, 313MHz 
OX  =  0, 807MHz 
juy  =  3,654MHz 
ay  =  1, 346MHz 

fi’x  =  0,071  MHz 
o’x  =  0,418MFIz 


On  peut  remarquer  la  correction  du  biais  introduit  un  ecart 
moyen  de  X  quasi  nul  et  un  ecart  type  moyen  de  0,418MHz 
qu’il  faut  comparer  a  l’ecart  moyen  de  Y et  a  l’ecart  type  moyen 
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dc  Y.  On  peut  noter  que  les  resultats  obtenus  sont  tout  a  fait 
bons  pour  ce  genre  de  mesures.  Un  ecart  type  de  0,4MHz  peut 
etre  considere  comme  la  precision  moyenne  des  sondeurs 
zenithaux.  La  methode  de  prevision  de  la  frequence  critique 
FoF2  apparait  done  tout  a  fait  correcte. 

6.  FUSION  DE  DONNEES  ET  GESTION 

L’analyse  du  spectre  decametrique  conduit  a  la  determination  de 
deux  parametres  importants  de  F  ionosphere  concemant  la 
gestion  des  provisions  des  interferences  et  la  provision  du 
parametre  foF2.  L’objectif  de  la  methode  ddcrite  est  d’etre 
totalement  passive  et  done  utilisable  par  tous  les  systemes  pour 
lesquels  unc  grande  discretion  est  requise.  Le  systeme 
fonctionne  alors  par  fusion  de  donnees  selon  le  modele  illustre 
sur  la  figure  8. 


Figure  8  :  Fusion  de  donnees  pour  previsions  passives 

L’ensembie  des  emetteurs  de  la  gamme  decametrique  existant  a 
la  surface  terrestre  constitue  la  banque  de  donnees  analysde  par 
un  recepteur  panoramique  qui  fournit  le  spectre  decametrique, 
rdsultat  de  la  fusion  des  donnOes  individuelles.  II  faut  distinguer 
dans  ces  donnees  les  bandes  de  radio-diffusion  des  autres 
bandes.  L’analyse  a  montre  que  1’ utilisation  de  ces  demieres 
donne  de  meilleurs  resultats  que  l’utilisation  des  bandes  de 
radio-diffusion.  Les  analyses  sont  ensuite  poursuivies  par  la 
segmentation  du  spectre  dans  les  95  bandes  allouees  auxquelles 
sont  attaches  4  parametres  qui  les  representent  completement. 
La  fusion  de  donnees  peut  alors  etre  completee  par  Pinsertion 
de  parametres  solaires,  de  parametres  magnetiques  et  d ’autres 
parametres  pouvant  etre  obtenus  par  des  methodes  totalement 
passives.  Ces  parametres  agissent  comme  variables  exogcncs 
dans  le  precede  de  traitement  des  fusions  de  donnees.  La  fusion 
de  donnees  fournit  alors  les  previsions  de  la  frequence  critique 
FoF2  de  l’ionosphere  et  la  provision  des  niveaux  des 
interferences.  Pour  effectuer  la  fusion  de  donnees  il  est 
preferable  d’utiliser  la  methode  de  la  theorie  des  possibility 
plutot  que  la  methode  de  la  theorie  des  probabilites  car  la 
theorie  des  possibility  introduit  une  variable  supplemental 
qui  permet  de  fiabiliser  plus  fortement  la  methode  de  provision. 

Les  resultats  obtenus  permettent  d’assurer,  sur  chaque  niveau 
d’interferences,  avec  un  preavis  de  l’ordre  de  4  a  5  heures,  des 


precisions  de  niveau  moyen  d’interferences  voisin  de  7dB  et 
une  precision  sur  la  frequence  foF2  de  l’ordre  de  0,4MHz.  Ces 
resultats  sont,  la  plupart  du  temps,  satisfaisants  pour  resoudre 
les  problemes  de  gestion  de  systemes  ne  pouvant  utiliser  des 
methodes  de  previsions  actives  telles  que  l’utilisation  de 
sondages  ionospheriques.  Les  performances  obtenues  fiabilisent 
fortement  l’etablissement  et  le  maintien  des  telecommunications 
d’autant  que  ces  systemes  peuvent  etre  dans  le  cas  de  liaisons  a 
courtes  distances  -  inferieures  a  1000km  -  implantes  aux  deux 
extremites  ce  qui  permet  de  faciliter,  entre  autres,  les  choix  de 
frequences  identiques  pour  realiser  la  liaison.  En  tout  etat  de 
cause  cette  methode  apporte  des  avantages  indeniables  par 
rapport  aux  methodes  de  prevision  a  long  terme. 

7.  CONCLUSION 

Les  interferences  constituent  dans  la  gamme  des  ondes 
decarndtriques  une  gene  tres  importante  pour  les 
telecommunications.  Une  modelisation  spatio-temporelle  des 
interferences  montre  qu’il  est  possible  d’etabiir  des  previsions 
pour  prevoir  au  jour  le  jour  leurs  evolutions.  Les  resultats 
obtenus  en  matiere  de  prevision  des  niveaux  des  interferences 
montrent  qu’il  est  possible  d’etabiir  une  modelisation 
suffisamment  precise  qui  permet  d’obtenir  des  previsions  a 
court  terme  avec  une  bonne  fiabilite.  L’operateur  dispose  done 
d’un  systeme  permettant  de  choisir  a  priori  la  frequence  qui 
restera  claire  pour  la  duree  de  la  liaison  qu’il  a  prevue. 
L’utilisation  du  spectre  decametrique  pour  l’etablissement  de 
prevision  de  propagation  basee  sur  Sexploitation  des  evolutions 
de  la  forme  du  spectre  apparait  possible  et  introduit  une 
nouvelle  methode  de  connexion  dans  les  systemes 
operationnels. 

La  rndthode  dlaboree  est  simple  et  son  experimentation  a  donne 
d’excellents  resultats.  L’ implantation  de  ce  systeme  de 
prevision  passive  de  transmission  des  canaux  ionospheriques 
apparait  possible  et  contribue  a  leur  evaluation  en  temps  reel. 
Testdes  sur  des  liaisons  courtes  distances  -  inferieures  a 
1000km  -  le  succes  de  ces  deux  procedures  a  ete  constate.  Son 
extension  a  de  grandes  distances  peut  etre  envisagee  mais  il 
apparait  que  la  decorrelation  spatiale  des  interferences  doit  etre 
corrigee.  Cette  correction  est  accessible  et  il  serait  alors 
particulicrement  interessant  d’introduire  les  resultats  de  ces 
mesures  dans  un  processus  de  correction  de  previsions  a  long 
terme  ou  de  programmes  universels  qui  permettrait  d’obtenir 
une  representation  de  l’ionosphere  et  des  interferences  a 
l’echelle  mondiale  et  par  des  methodes  totalement  passives. 
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1.  SUMMARY 

Radiocommunications  in  decametric  band  are  made  difficult  by 
important  variability  of  transmission  channel  and  by  level  and 
density  of  interferences  which  congest  this  part  of  the 
radioelectric  spectrum. 

Link  establishment  methods  are  currently  based  on  a 
preliminary  sounding  of  the  channels  and  a  scrutation  of 
interference  spectrum.  This  method  is  implemented  by  systems 
which  works  alternately  with  radiocommunication  systems  or 
use  independent  test  system. 

The  disadvantages  of  these  methods  lie  in  an  uneasy  uses  and  a 
often  insufficient  periodic  actualisation  of  estimation. 

Implementation  of  system  allowing  to  test  channel  with 
extremely  low  emitted  power  gives  a  new  way  very  interesting 
to  establish  links  and  to  keep  them. 

Electromagnetic  soundings  of  the  ionosphere  is  limited  by  the 
problem  of  electromagnetic  pollution  they  create,  which  limits 
their  use,  and  by  the  high  sensitivity  of  the  interference  included 
in  the  band.  The  emitted  power  reduction  solves  the  first  point, 
on  condition  that  sensitivity  to  the  jammers  remains  acceptable. 

This  results  is  obtained  for  implementation  of  non  linear  signal 
processing  which  allows  to  choose  the  wave  shape  and  coding 
by  solving  an  optimisation  problem  without  constraints.  Results 
obtained  allows  to  do  soundings  with  very  low  power.  Figure  1 
presents  a  zenithal  sounding  measured  with  a  emitted  power  of 
lOmW. 

The  establishment  of  this  ionogram  with  the  used  method  takes 
26  seconds.  Figure  2  and  3  show  results  obtained  with  a 
backscattering  sounding  and  compare  a  classical  method  with 
the  new  method  used,  code  time  duration  used  and  emitted 
power  are  identical  in  the  two  cases. 

-  On  figure  2,  an  impulse  response  is  represented. 

-  On  figure  4,  a  scattering  function  is  represented. 

We  can  notice  the  protection  given  by  the  new  method. 

These  news  capacities  allow  a  permanent  ionospheric  sounding, 
on  all  the  frequencies.  A  data  fusion  can  then  be  made  taking 
into  account  available  informations  (figure  7),  especially 
ionospheric  disturbances. 

This  method  can  be  applied  in  particular  to  the  network 
management  to  improve  their  efficiency  (figure  8). 

2.  GENERALITES 

Les  radiocommunications  dans  la  gamme  decametrique  sont 
rendues  difficiles  par  les  grandes  variations  des  caracteristiques 
du  canal  et  par  le  niveau  et  la  densite  des  interferences  qui 
encombrent  cette  partie  du  spectre  radioelectrique. 


Les  methodes  d’etablissement  des  liaisons  sont  actuellement 
basees  sur  un  test  preliminaire  des  canaux  et  une  scrutation  du 
spectre  des  interferences.  Ces  methodes  mettent  en  ceuvre  des 
systemes  de  test  separes.  Les  inconvenients  de  ces  methodes 
sont  lies  aux  lourdeurs  d’utilisation  et  a  la  reactualisation 
periodique  souvent  insuffisante  des  estimations. 

La  mise  en  oeuvre  de  systemes  permcttant  de  tester  les  canaux 
avec  des  puissances  d'emission  extremement  faibles,  donne  une 
dimension  nouvelle  particulicrement  interessante  pour 
l’etablissement  des  liaisons  et  leur  maintien. 

Les  sondages  clectromagnctiques  de  F ionosphere  se  heurtent  au 
probleme  de  la  pollution  electromagnetique  qu’ils  creent,  ce  qui 
limite  leur  utilisation,  et  a  leur  sensibilite  aux  interferences 
presentes  dans  la  bande.  La  reduction  de  la  puissance 
d’emission  resout  le  premier  point  a  condition  que  la  sensibilite 
aux  brouillages  demeure  acceptable. 

Ce  resultat  est  atteint  par  Tutilisation  de  traitement  de  signaux 
non  lineaires  qui  permettent  de  choisir  la  forme  d’onde  et  le 
codage  par  la  resolution  d’un  probleme  d’optimisation  sous 
contraintes. 

II  est  present^,  dans  ce  document,  les  resultats  de  methodes  de 
sondages  electromagnetiques  de  l’ionosphere  associant  des 
durees  d’analyse  tres  courtes  a  des  puissances  emises  tres 
faibles.  Partant  de  ces  resultats,  il  a  ete  envisage  une  methode 
permettant  d’effectuer  des  sondages  permanents  de  Tionosphere 
sans  entrainer  de  pollution  electromagnetique  du  spectre 
decametrique,  ce  qui  permet  d’envisager  des  methodes 
d’analyse  de  canaux  qui,  associees  a  des  methodes  de  prevision 
d’interferences,  permettent  une  gestion  optimale  beaucoup  plus 
sure  que  par  les  methodes  conventionneliement  utilisees.  Ces 
methodes,  particulierement  economiques  car  elles  necessitent 
des  puissances  tres  reduites,  conduisent  a  des  systemes  de 
sondage  indetectables  auxquels  il  ne  peut  done  etre  reproche  de 
gener  les  autres  utilisateurs  [1]  [2]. 

Le  systeme  envisage  consiste  a  allier  le  sondage  zenithal  de 
l’ionosphere  et  une  methode  de  sondage  par  retrodiffusion  tout 
azimut  qui  permet  de  dresser,  a  partir  d’un  point  unique  situe  au 
sol,  une  cartographie  de  Tionosphere  et  de  connaitre  ainsi  sur 
une  zone  de  6000km  de  diametre  les  modes  de  propagation 
pouvant  exister  entre  deux  points  quelconques  de  la  zone. 
L’information  apportee  par  la  mesure  de  la  fonction  de  diffusion 
permet  egalement  de  prendre  en  compte  les  effets  dynamiques 
du  milieu  et  d’assurer  ainsi  une  meilleure  fiabilite  des  systemes 
de  telecommunication  haute  frequence  par  voie  ionospherique. 
La  methode  proposee  s’applique  a  tous  les  types  de  sondages  de 
l’ionosphere,  verticaux,  par  retrodiffusion  ou  obliques.  Dans 
cette  derniere  utilisation,  le  principe  de  la  gestion  de  reseau  doit 
etre  considere  comme  une  extension  des  methodes  actuelles. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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3.  SONDAGES  ELECTROMAGNETIQUES  DE 
L’lONOSPHERE  AVEC  DES  SYSTEMES  A 
EXTREMEMENT  FAIBLE  PUISSANCE 

II  est  bien  connu  que  les  resultats  de  sondages 
electromagnetiques  qui  s’appuient  sur  les  techniques  radars 
peuvent,  par  Tutilisation  de  sequences  codees,  foumir  des 
resultats  de  grande  qualite  pourvu  que  l’energie  consacrde  a  une 
mesure  soit  suffisante. 

La  technique  de  compression  d’intpulsions,  utilisee  dcpuis 
longtemps,  a  permis  de  reduire  les  puissances  des  radars  dans 
un  facteur  de  plusieurs  dizaines.  L’utilisation  dc  la  technique  de 
remission  continue  par  modulation  lineaire  de  frequence  - 
technique  «  CHIRP  »  -  a  permis  d’effectuer  des  sondages  avec 
des  puissances  tres  faibles  mais  cette  technique  necessite  des 
systemes  bistatiques. 

En  1968,  le  LETTI  a  propose  une  methode  dite  d’impulsions 
longues  codees  qui  a  consiste  a  repartir  des  codes,  non  plus  4 
l’interieur  d’une  impulsion  comme  dans  le  cas  des  systemes  a 
compression  d’impulsions,  mais  sur  un  ensemble  d’impulsions 
consecutives  de  fa?on  a  pouvoir  conserver  les  proprietes  des 
systemes  monostatiques  et  a  utiliser  des  sequences  tres  longues. 
Cette  methode  a  donne  rapidement  de  tres  bons  resultats  mais 
elle  a  ete  limitee  par  l’instabilite  du  milieu  de  propagation  qui 
fixe  une  borne  superieure  a  la  sequence  et,  par  consequent,  en 
limite  les  performances.  Les  applications  de  cette  methode  ont 
donne  lieu  a  la  mise  en  oeuvre  de  la  station  STUDIO  (Systeme 
de  Traitement  Universel  de  Diagnostic  IOnospheriques)  du 
LETTI  en  1968  et  des  applications  de  cette  technique  ont  etc 
reprises  dans  le  radar  transhorizon  frangais  NOSTRADAMUS 
(Nouveau  Systeme  TRAnshorizon  Decametrique  Appliquant 
les  Methodes  Utilisees  dans  Studio).  Une  station  a  dgalement 
ete  construite  au  C.N.E.T.  (Centre  National  d’Etude  des 
Telecommunications)  dans  laquelle  le  codage  et  le  traitement  de 
signal  de  la  station  STUDIO  ont  ete  introduits  des  1985. 

Les  progres  dans  l’efficacite  des  methodes  de  sondage  de 
l’ionosphere  ne  pouvaient  plus  des  lors  s’appuyer  sur  ces 
techniques  qui  avaient  ete  testees  durant  de  longues  annees. 
Deux  voies  ont  ete  developpees  au  LETTI  et  ont  conduit  a  des 
avancees  significatives  dans  le  domaine  de  la  reduction  des 
puissances  utilisables.  La  premiere  a  consiste  a  rechercher  des 
sequences  mieux  appropriees  aux  objectifs  vises  et  la  seconde  a 
incorporer  dans  le  traitement  de  signal  des  methodes  non 
lineaires. 

3.1.  Sequences  adaptees  aux  techniques  de  sondage 

L’utilisation  des  techniques  de  codage  par  (’utilisation  de 
sequences  emises  pour  obtenir  des  taux  de  compression  eleves 
est  basee  essentiellement  sur  l’analyse  des  proprietes  de  la 
fonction  de  diffusion  du  signal  utilise.  Les  caracteristiques 
generales  de  la  fonction  de  diffusion  sont  bien  connues,  le 
volume  enserre  par  le  carre  du  module  est  une  constante  et  la 
fonction  de  diffusion  est  maximale  pour  une  frequence  Doppler 
nulle  et  un  decalage  temporel  nul.  La  fonction  de  diffusion 
permet  de  caracteriser  les  proprietes  d'un  signal  en  presence  de 
bruit  gaussien  mais  en  presence  d’ interferences,  dont  les 
caracteristiques  ne  sont  pas  gaussiennes,  les  caracteristiques  de 
la  fonction  de  diffusion  ne  sont  plus  suffisantes.  La  reduction  de 
ia  puissance  emise  s’appuie  sur  la  prise  en  compte  des 
paramhtres  suivants  : 

Les  proprietes  d’autocorrelation  de  la  sequence  utilisee  qui 
doivent  rester  excellentes,  c’est-a-dire  proches  de  la 
fonction  de  correlation  parfaite  qui  prend  des  valeurs  nulles 
en  dehors  du  pic  central. 


La  fonction  de  diffusion  doit  presenter  une  forme  non 
homogene  en  dehors  du  pic  central  afin  d’apporter  une 
protection  maximale  dans  la  zone  correspondant  au 
Doppler  dont  peut  etre  affecte  le  signal  utile. 

La  sequence  doit  avoir  des  caracteristiques  qui  entralnent, 
vis-a-vis  des  interferences  qui  constituent  les  signaux  les 
plus  genants  de  la  gamme  decametrique  des  protections 
maximales. 

Concernant  la  premiere  caracteristique,  il  apparait  que  les 
sequences  classiquement  utilisees,  tels  que  les  codes  PN  -  ou 
codes  de  Huffman  -  n’apportent  pas  de  protection  suffisante  sur 
la  fonction  de  correlation  et  entralnent  des  biais  dans  les 
signatures  des  sondages  de  I ’ionosphere.  Les  sequences  de 
Barker  apparaissent  trop  courtes  et  les  sequences  de  Golay 
necessitent  une  emission  impulsionnelle  incompatible  avec  des 
taux  de  compression  tres  eleves.  II  est  interessant  d’utiliser  des 
sequences  longues  ayant  des  fonctions  de  correlation  aussi 
proches  que  possible  que  de  la  correlation  parfaite.  II  a  ete 
demontre  [3]  qu’il  n’existe  pas  de  sequences  binaires  ayant  des 
fonctions  de  correlation  parfaites  et  que  les  meilleures 
sequences  devaient  admettre,  en  dehors  du  pic  central  de 
correlation,  un  pic  secondaire  situe  au  milieu  de  la  plage  nulle 
de  la  fonction  de  correlation  periodique  de  la  sequence.  Ces 
sequences  qui  ont  ete  determinees  [3]  ont  ete  appelees 
sequences  W.G.  et  il  est  montre  que  ces  sequences  existent  pour 
de  tres  nombreuses  longueurs  multiples  de  4  qui  ont  ete 
determinees  jusqu’a  des  longueurs  20  000.  Pour  chaque 
longueur  il  existe  egalement  une  grande  variete  de  sequences 
dont  les  proprietes  sont  decrites. 

En  ce  qui  concerne  les  sequences  Q-aires,  des  sequences  telles 
que  les  sequences  de  Franck  ont  dte  trouvees.  Cependant,  ces 
sequences  apparaissent  comme  un  cas  particulier  des  sequences 
plus  generales  qui  ont  ete  publiees  dans  [4]  et  qui  presentent  des 
conditions  d’orthogonalite  particulierement  utiles  en  matiere  de 
protection  vis-a-vis  des  brouilleurs  et  des  problemes  de 
detection  en  general  et  une  methode  de  determination  de 
sequences  plus  generale  a  ete  proposee  dans  [5]  et  a  conduit  a 
une  technique  de  determination  generale  qui  permet  de  prendre 
en  compte  des  contraintes  supplementaires  telles  que  les 
caracteristiques  generales  des  interferences.  Ces  techniques  ont 
ete  mises  en  oeuvre  pour  determiner  un  ensemble  de  sequences 
particulierement  adaptees  au  sondage  electromagnetique  de 
I’ionosphere. 

3.2.  Traitement  des  signaux 

La  seconde  voie  qui  a  ete  utilisde  pour  reduire  les  puissances 
d’emission  concerne  les  traitements  lineaires  susceptibles  d’etre 
mis  en  ceuvre.  Des  traitements  non  lineaires  telle  que  la 
technique  d ’excision  ont  ete  proposes  des  le  debut  des  annees 
1980  [6]  ct  donnent  des  resultats  satisfaisants  dans  le  cas  oh 
l’occupation  spectrale  du  brouilleur  est  faible  devant  le  domaine 
frdquentiel  occupe  par  le  signal  utile.  D’autres  techniques  non 
lineaires  plus  puissantes  peuvent  etre  utilisees  comme 
1’utilisation  des  moments  d’ordre  superieur,  mais  leur  mise  en 
ceuvre  pratique  se  heurte  au  volume  de  calculs  necessaires  pour 
les  utiliser.  Bien  que  des  algorithmes  visant  a  reduire  la 
complexite  des  calculs  aient  ete  proposes,  il  apparait  encore  tres 
difficile  de  songer  a  les  appliquer  dans  des  systemes 
operationnels.  Des  concepts  nouveaux  peuvent  etre  mis  en 
ceuvre  en  s’appuyant  sur  la  definition  d’un  processus  non 
lindaire  defini  par  sa  reponse  impulsionnelle 
multidimensionnelle  et  qui  permet  de  rendre  compte  de  la 
totalite  du  processus  et  egalement  de  le  definir.  A  partir  de  ce 
concept,  il  devient  possible  d’examiner  la  reponse  d’un  systeme 
non  lineaire  aux  differents  types  d’ interferences  rencontres  dans 
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le  spectre  decametrique.  Le  processus  peut  alors  etre  optimise 
par  la  resolution  d’un  probleme  d’optimisation  sous  contraintes. 
Le  probleme  se  reduit  a  rendre  maximum  un  parametre  -  qui 
peut  etre  le  rapport  signal  a  bruit  en  sortie  du  processus  -  sous 
la  contrainte  de  [’utilisation  d’un  signal  utile  constitue  par  une 
sequence  ayant  de  bonnes  caracteristiques  en  presence  de 
signaux  parasites  constitues  par  les  differents  types 
d’interferences. 

Les  processus  non  lineaires  peuvent  alors  etre  determines  par  la 
resolution  informatique  de  ces  problemes  d’optimisation. 
L’utilisation  de  processus  non  lineaires  offre  sur  les  processus 
lineaires  l’avantage  considerable  d’offrir  des  structures 
extremement  diverses  qui  permettent  d’obtenir  une  optimisation 
plus  efficace  dans  un  cadre  plus  general. 

4.  RESULTATS  OBTENUS  AVEC  LES  SONDEURS  A 
EXTREMEMENT  FAIBLE  PUISSANCE 
Les  performances  obtenues  par  les  techniques  mises  en  oeuvre 
sont  illustrees  par  un  ensemble  d’exemples  presentes  dans  ce 
paragraphe.  Ces  techniques  sont  actuellement  utilisees  dans  les 
systemes  de  sondages  zdnithaux,  obliques  bistatiques  et  par 
retrodiffusion.  Deux  types  de  resultats  sont  presentes,  l’un 
concemant  le  sondage  zenithal,  l’autre  le  sondage  par 
retrodiffusion. 

Le  sondage  zenithal  est  une  mesure  qui  est  particulierement 
bien  connue  et  les  techniques  utilisdes  ont  etd  frequemment 
decrites.  Dans  les  systemes  monostatiques  les  sequences  de 
Golay  sont  couramment  utilisees  pour  realiser  le  sondage.  On  a 
compare  les  resultats  obtenus  dans  l’utilisation  des  sequences  de 
Golay  avec  les  resultats  obtenus  en  utilisant  la  technique 
proposee.  On  a  utilise  pour  effectuer  la  comparaison  deux 
mesures  permettant  d’effectuer  des  sondages  avec  la  meme 
duree.  La  figure  1  represente  les  resultats  obtenus  avec  ces  deux 
methodes.  L’ionogramme  de  la  figure  la  a  etc  obtenu  en 
utilisant  des  sequences  de  Golay  a  16  moments  avec  une 
puissance  d’emission  de  150W.  La  duree  de  mesure  est  de  26 
secondes  et  on  peut  voir  apparaTtre  les  echos  obtenus  sur  la 
region  E  et  sur  la  region  F.  On  peut  noter  que  le  mode  2F  est 
parfaitement  visible.  La  figure  lb  represente  l’ionogramme 
obtenu  le  meme  jour  avec  la  methode  proposee.  La  puissance 
d’emission  a  ete  reduite  a  lOmW  et  la  duree  de  mesure,  avec  la 
technique  proposde,  est  reduite  a  22  secondes.  On  peut  noter 
que  la  region  E  et  la  region  F  sont  parfaitement  visibles  et  que 
les  interferences  presentes  sur  l’enregistrement  effectue  avec 
des  codes  de  Golay  sont  fortement  attcnuces.  Ces  resultats 
illustrent  la  possibility  d’effectuer  des  sondages  zenithaux  avec 
de  tres  faibles  puissances. 

Figure  1  :  Comparaison  des  performances  des  ionogrammes 
zenithaux  obtenus 


GOLAY  SEQUENCES  IONOGRAM 


Figure  La  :  1501V  Code  de  Golay,  Temps  de  mesure  :  26s 
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figurel.b  :  Nouvelle  methode  code  non  systematique  10  mW 
STUDIO  1997  Temps  de  mesure  :  22s 

La  meme  methode  est  utilisee  pour  effectuer  des  sondages  par 
retrodiffusion  qui  exigent  des  puissances  superieures  a  celles 
utilisees  dans  les  sondages  zenithaux.  Le  principe  du  sondage 
par  retrodiffusion  s’appuie  en  effet  sur  la  technique  radar 
appliqude  a  des  distances  de  plusieurs  milliers  de  kilometres 
alors  que  le  sondage  zenithal  peut  etre  considere  comme  une 
transmission  point  a  point  la  reflexion  sur  l’ionosphere  etant  du 
type  speculaire.  Les  differences  de  bilans  de  propagation  sont  de 
l’ordre  de  120  a  140dB  ce  qui  necessite,  pour  le  sondage  par 
retrodiffusion.  des  puissances  habituellement  beaucoup  plus 
elevees  que  dans  les  sondages  zenithaux.  Dans  les  sondages  par 
retrodiffusion  les  puissances  ont  progressivement  diminue  de 
plusieurs  centaines  de  kilowatts  initialement  utilisees  a  quelques 
kilowatts  dans  les  methodes  actuelles.  La  nouvelle  methode 
permet  d’obtenir  des  resultats  avec  des  puissances  plus  reduites. 

La  figure  2  illustre  les  performances  obtenues  en  retrodiffusion. 
La  figure  2a  montre  un  echo  de  retrodiffusion  s’etendant 
jusqu’a  7500km  pour  une  puissance  d’emission  de  300W.  La 
figure  2b  represente  une  evolution  de  la  reponse  impulsionnelle 
du  canal  sur  une  duree  de  30  minutes.  Ces  resultats  ont  ete 
obtenus  par  une  methode  deja  elaboree  dans  les  annees  1990  et 
qui  a  ete  depuis  amelioree.  La  figure  2c  represente  une  reponse 
impulsionnelle  obtenue  avec  une  puissance  d’emission  de 
lOOmilliwatts  et  la  figure  2d  represente  F  evolution  temporelle 
de  cette  reponse  impulsionnelle  sur  une  duree  de  30  minutes. 
Ces  resultats  ont  dte  obtenus  avec  l’utilisation  d’une  antenne 
log-periodic  dont  le  gain,  dans  la  direction  du  lobe  principal  est 
de  l’ordre  de  9dB.  II  va  de  soi  qu’associer  a  un  reseau  les 
resultats  sont  ameliores  de  fa?on  sensible. 


Figure  2  :  Mesures  en  retrodiffusion 


Figure2.a  et  2.b  :  Nouvelle  methode  300W 1988 
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Figure  2.c  et  2.d  :  Nouvelle  methode  0.1  W  STUDIO  1988 


La  figure  3  permet  de  juger  de  Fefficacite  des  sequences  mises 
en  oeuvre.  Pour  tester  les  sequences,  on  a  effectue  sur  une 
periode  de  12  minutes  deux  essais  consecutifs  en  conservant  la 
puissance  d’emission  invariante  et  sans  modifier  les  reglages  de 
l’emetteur  et  du  recepteur  de  la  station.  La  frequence  d’emission 
a  ete  choisie  de  fafon  a  ce  que  la  mesure  soit  perturbee  par  des 
interferences  genant  la  reception.  Durant  les  six  premieres 
minutes,  on  a  utilise  une  sequence  pseudo-aleatoire  de  longueur 
4095.  On  peut  constater  que,  durant  cette  periode,  le  brouillage 
est  complet  et  qu’aucune  trace  de  l’echo  de  retrodiffusion  n’est 
decelable.  Durant  les  six  demieres  minutes  on  a  utilise  une 
sequence  nouvelle  appelee  code  non  systdmatique.  La  longueur 
de  la  sequence  utilisee  est  de  3937,  c’est-a-dire  extremement 
voisin  de  celle  de  la  sequence  pseudo-aldatoire  bien  qu'on 
puisse  noter  que  les  longueurs  choisies  avantage  cette  derniire. 
On  peut  noter,  durant  ces  six  demieres  minutes,  l’apparition  de 
1’echo  de  retrodiffusion  et  une  rejection  totale  des  interferences 
apparaissant  dans  cette  periode.  II  convient  de  noter  que  cette 
illustration  met  en  evidence  1’ importance  du  codage,  les 
traitements  de  signaux  etant  strictement  identiques  pour  la 
sequence  pseudo-aleatoire  et  pour  la  sequence  non 
systematique. 


Figure  3  :  Protection  contre  les  interferences  avec  methode  de 
codage  classique  et  nouveau  codage  utilisant  un  code  non 
systematiques 

La  figure  4  illustre  enfin  les  resultats  obtenus  pour  la  mesure  de 
fonction  de  diffusion.  On  a  mesure  a  la  meme  frequence  et  de 
fa?on  immcdiatement  consecutive  deux  fonctions  de  diffusion, 
l’une  par  l’utilisation  d’une  sequence  pseudo-aleatoire,  l’autre 
par  [’utilisation  d’une  sequence  non  systematique.  On  peut  noter 
la  presence  d’un  brouilleur  qui  rend  invisible  l’echo  de 
retrodiffusion  dans  lc  cas  de  la  detection  par  une  sequence 
pseudo-aleatoire.  Sur  la  figure  inferieure,  on  peut  noter  que  le 


changement  de  sequence  permet  de  retrouver  l’echo  de 
retrodiffusion.  II  faut  signaler  encore  que  dans  cette 
experimentation,  les  rdglages  de  l’emetteur  et  du  recepteur  ont 
ete  invariants. 

Les  resultats  presentes  donne  une  illustration  des  performances 
atteintes  actuellement  par  les  nouvelles  methodes  mises  en 
oeuvre. 

Figure  4  :  Effets  des  interferences  sur  la  fonction  de  diffusion 
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Figure  4. a  :  Code  pseudo-aleatoire 
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Figure  4.b  :  Code  non  systematique 

Le  tableau  1  resume  une  Evolution  des  puissances  utilisees 
habituellement  pour  effectuer  differents  types  de  sondages  : 
vertical,  oblique  bistatique  et  par  retrodiffusion.  Avec  les 
nouvelles  methodes  presentees,  les  puissances  pouvant  etre 
utilises  dans  des  systemes  operationnels  sont  de  l’ordre  du  watt 
pour  les  sondages  verticaux  et  obliques  bistatiques  et  de  l’ordre 
de  50  a  100  watts  pour  les  sondages  par  retrodiffusion.  Ces 
ordres  de  grandeur  ouvrent  de  nouvelles  possibility  pour  la 
realisation  de  systemes  de  gestion  automatique  des 
telecommunications  dans  la  bande  decametrique. 


YEARS 

VERTICAL 

OBLIQUE 

BISTATIC 

BACK- 

SCATTERING 

m 

10  kW 

10  kW 

100  kW 

■Ml 

I  kW 

1  kW 

100  kW 

1980’s 

100  W 

100  W 

10  kW 

1997 

1  W 

1  W 

100  W 

(lOmW) 

(lOmW) 

(0,1  W) 

Tableau  1  :  Evolution  des  puissances  utilisees 


5.  APPLICATION  AUX  TELECOMMUNICATIONS. 
GESTION  DE  RESEAUX 

Compte  tenu  des  resultats  qui  peuvent  actuellement  etre  utilises, 
on  a  considere  que  la  gestion  des  reseaux  pouvait  etre  amelioree 
par  une  utilisation  plus  frequente  des  sondeurs.  Les  hypotheses 
prises  pour  concevoir  un  systeme  de  gestion  de  reseaux  en 
temps  reel  sont  les  suivantes  : 

Les  puissances  utilises  par  les  sondeurs  -  typiquement  1W 
pour  les  sondages  zenithaux  et  obliques  bistatiques,  50  a 
100W  pour  les  sondages  par  retrodiffusion  -  sont 
suffisammcnt  faibles  pour  que  les  sondeurs  puissent  etre 
utilises  sans  apporter  de  nuisances  aux  autres  utilisateurs  et 
sans  etre  detectables  par  ceux-ci.  On  peut  remarquer  que 
les  sondages  par  retrodiffusion  qui  utilisent 
traditionnellement  des  largeurs  de  bandes  de  plusieurs 
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dizaines  de  kHz  conduisent  a  des  densites  de  puissances 
extremement  faibles  par  bandes  de  3kHz.  A  titre 
d’exemple,  l’utilisation  d’une  puissance  de  60W  avec  un 
signal  dont  le  spectre  s’etend  sur  30kHz  conduit  a  une 
densite  de  puissance  emise  de  6W  pour  3kHz  ce  qui  rend 
son  emission  indetectable  par  des  systemes  classiques. 

Le  systeme  de  prevision  est  un  systeme  centralist  utilisant 
un  sondage  zenithal  et/ou  un  sondage  par  retrodiffusion. 
Des  sondages  obliques  bistatiques  peuvent  etre  utilises  en 
complementarity  avec  les  sondages  precedents  ou  bien  de 
fa9on  isolee  a  condition  que  ces  sondages  soient  inclus 
dans  les  systemes  de  telecommunication. 


Nous  nous  interesserons  dans  cette  communication  a  la  gestion 
centralist  d’un  systeme  utilisant  les  sondages  zenithaux  et  par 
retrodiffusion.  On  considerera  que  le  sondage  zenithal  donne 
une  information  de  bonne  precision  sur  le  profil  d’ ionisation  a  la 
verticale  de  la  station.  Le  sondage  par  retrodiffusion  foumit  les 
cinq  types  d’enregistrements  (figure  5) :  la  fonction  de  diffusion 
du  canal,  de  l’ionogramme  de  retrodiffusion,  une  representation 
PPI  de  la  propagation  et  les  propagations  longues  distances 
faisant  apparaitre,  en  particulier,  les  super-modes.  Ce  type  de 
sondage  offre  bcaucoup  d’interet  car  il  permet  de  mettre  en 
evidence  l’anisotropie  du  milieu  ionospherique.  de  faire 
apparaitre  directement  les  MUF  et  donner  acces  a  revaluation 
des  LUF  ainsi  que  par  1’ intermediate  de  la  fonction  de  diffusion 
et  de  la  representation  PPI  de  la  detection  des  perturbations 
ionospheriques,  de  leur  evolution  et  de  leur  prevision. 


Scattering  function 
Ionospheric  disturbance 

\ 


Long  distance 
propagation 


•Ionogram  all  azimuth  =  PPI  all  frequency 
•Long  distance  propagation 
•Ionospheric  disturbances 
•Time  evolution 


Figure  5  :  Donnees  issues  du  sondage  par  retrodiffusion 


L’ensemble  de  ces  mesures  est  alors  traite  par  une  methode  de 
fusion  de  donnees  dans  laquclle  les  grandeurs  d’entrde 
sont  (figure  7)  : 

Les  sondages  zenithaux,  les  sondages  obliques  bistatiques, 
les  sondages  par  retrodiffusion. 

Les  donnees  relatives  a  l’activite  solaire. 

Les  donnees  relatives  a  Factivite  magndtique. 

D’autres  mesures,  telles  mesures  pouvant  provenir  de 
satellites,  de  reseaux  de  sondeurs,  de  mesures 
ionospheriques  diverses. . . 


Solar 

Data 


Mag^ikk  i 
^  Bate 


Real  Time  ierKiapbeii 
;  mods! 


Channel  communications 
Parameters 


Figure  6  :  Fusion  de  donnees  pour  une  station  de  sondage 
ionospherique  unique 

La  fusion  de  donnees  realise  alors  une  reactualisation  du  modele 
ionospherique  par  la  prise  en  compte  de  toutes  les  donnees 
disponibles  pour  foumir  un  modele  ionospherique  temps  reel 
sur  une  zone  de  6000km  de  diametre.  Les  resultats  sont  alors 
presentes  par  des  cartes  telle  que  celle  apparaissant  sur  la  figure 
8  permettant  de  determiner  les  proprietes  du  canal  pour  des 
liaisons  s’effectuant  soit  entre  un  point  voisin  de  la  station  et  un 
correspondant  situe  dans  la  zone,  soit  entre  deux  correspondants 
situes  dans  cette  zone.  Le  principe  consiste,  pour  etablir  ces 
liaisons,  en  des  traces  de  rayons  obtenus  a  partir  du  modele 
ionospherique  temps  reel  reactual isc  par  la  fusion  des  donnees 
disponibles.  La  fusion  de  donnees  peut  etre  traitee  par 
differentes  methodes.  L’une  des  methodes  les  plus  classiques 
consiste  a  utiliser  la  theorie  des  probabilites.  II  est  apparu,  lors 
de  1 ’analyse  de  ce  probleme,  que  de  meilleurs  resultats  etaient 
obtenus  en  utilisant  la  theorie  des  possibiiites.  Cette  theorie 
presente  sur  la  theorie  des  probabilites  l’avantage  d’introduire 
une  variable  supplemental  qui  permet  une  solution  plus 
complete  et  plus  precise  du  probleme. 
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Figure  7  :  Exemple  d' application  a  la  gestion  dune  reseau 

La  methode  de  la  theorie  des  possibiiites  utilisee  peut  se 
re sumer  de  la  fa?on  suivante  : 
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1)  On  definit  un  ensemble  flou  par 

0,1] 

A  est  en  ensemble  flou  defini  sur  le  domaine  A  avec 

a) 

Vast/ 

Mao 

V-A,)  =  1  “  ^.) 

ou  ^  designe  la  mesure  du  contraire  de^^j 

b) 

Ac  B  implique 
VeU  ’ 

c) 

V„eU  ,  Vv  s  V 
Baxb(u,v)  ~  min(i/^(1,)  ’Bb(v)  ) 

2)  fj  est  defini  par  une  forme  trapezo'idale  symetrique 
qui  a  pour  support  fl(uj  >  0 

dont  le  noyau  est  defini  par  fl(„)  =  1 

Le  support  et  le  noyau  definissent  totalement  jd(u)  qui  est 

determine  pour  chaque  evenement. 

3)  La  resolution  du  probleme  se  reduit  alors  a  la  recherche  de 
la  probability  maximale  de  la  prevision  effectuee. 

L’application  de  cette  methode  conduit  a  des  previsions  tres 
precises  qui  permettent,  avec  des  temps  de  preavis  de  1  a  2 
heures,  de  determiner  les  frequences  optimales  avec  des 
probability  superieures  a  99%. 

5.  CONCLUSION 

La  possibility  d'effcctuer  des  sondages  de  Fionosphere  avec  des 
puissances  extremement  faibles  conduit  a  un  nouveau  concept 
de  gestion  de  reseaux  en  temps  reel.  Les  techniques  usuelles 
devaluation  en  temps  reel  des  canaux  (RTCE)  sont  souvent 
appliquees  dans  des  systemes  de  sondage  point  a  point  par 
Futilisation  des  equipements  de  transmission.  Par  rapport  a  cette 
methode  de  conception  deja  ancienne,  la  methode  proposee 
offre  Favantage  d’une  scrutation  permanente  de  Fionosphere 
avec  des  systemes  non  polluants,  indctectables  qui  peuvent  etre, 
soit  incorpores  dans  des  equipements  de  telecommunication, 
soit  fonctionner  de  fa?on  autonome  et  parallele.  Les  avantages 
lids  a  cette  methode  tiennent  essentiellement  au  fait  que  la 
scrutation  peut  etre  faite  de  fafon  adaptative  par  Foperateur 
dans  toutes  les  gammes  de  frequences  ce  qui  n’est  pas  toujours 
possible  dans  les  systemes  RTCE.  La  dimension  spatiale 
introduite  par  le  sondage  par  retrodiffusion  qui  devient  une 
methode  legere  compte  tenu  des  faibles  puissances  mises  en  jeu 
et  par  la  simplicity  des  aeriens  pouvant  etre  utilises  -  une  simple 
antenne  log-periodic  suffit  -  permet  une  gestion  a  partir  d’un 
site  unique  avec  une  prevision  spatiale  des  perturbations  qui 
transitent  a  travers  Fionosphere. 

L’avance  apportee  par  Futilisation  de  puissances  extremement 
faibles  permet  d’echapper  aux  critiques  justement  faites  dans  le 
passe  de  systemes  trop  polluants  ou  trop  onereux  pour  etre 
utilises.  II  apparait  desormais  que  ces  systemes  peuvent  etre 
incorpores  a  faible  cout  et  sans  pollution  dans  les  equipements 
futurs.  Les  progres  effectues  dans  Favenir  sur  les  traitements  de 
signaux  et  sur  les  techniques  de  codage  permettront 


probablement  de  rendre  ces  methodes  encore  plus  attractives  et 

contribueront  a  augmenter  la  fiabilite  des  telecommunications. 
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1.  SUMMARY 

A  new  method  of  SSL  (Single  Site  Location)  is  presented 
which  makes  use  of  multi-sensor  information.  Also,  unlike 
conventional  SSL  methods  which  generally  ray-trace 
backwards  using  a  single  angle  of  incidence  and  azimuth, 
this  method  uses  the  incident  angles  of  tw  o  or  more 
simultaneous  paths  (e.g.  1  and  2  hop  F2)  to  determine  both 
an  updated  ionosphere  model  at  the  reflection  point(s)  and 
the  location  of  the  target  transmitter.  The  angles  of 
azimuth  of  arrival  and  elevation  are  determined  using  a 
super-resolution  algorithm  to  decompose  the  multipath 
wavefield  received  on  an  antenna  array.  Analytic  models 
are  used  to  approximate  the  ionosphere  and  determine 
propagation  paths.  If  vertical  ionospheric  sounder  data  is 
available,  the  obtained  electron  density  profile  can  be 
fitted  using  quasi-parabolic  segments  (MQP  model)  and 
parameters  of  this  model  optimised  in  the  minimisation 
procedure.  A  Nelder  Mead  [1]  simplex  algorithm  is  used 
for  the  function  minimisation.  Greater  accuracy  in 
unknown  transmitter  location  can  be  achieved  by 
including  additional  constraints  in  the  system  such  as 
measured  elevation  angles  of  known  transmitters. 

The  SSL  determination  is  performed  under  MATLAB  via 
a  menu-driven  graphical  user  interface.  This  employs  pull¬ 
down  menus  for  data  input,  program  choice,  ionospheric 
profile  plotting  and  presentation  of  results. 

2.  INTRODUCTION 

The  SSL  method  enables  the  location  of  a  distant  HF 
transmitter  to  be  determined  from  a  single  site.  This  avoids 
the  need  for  employing  two  or  more  receiving  sites  and  the 
additional  complication  of  communications  between  them. 
Further,  for  the  case  when  the  base  line  for  triangulation  is 
short,  it  can  even  determine  more  accurate  transmitter 
locations  than  triangulation.  The  main  source  of  error  in 
the  conventional  HF  transmitter  SSL  technique  has 
generally  been  found  to  be  the  characterisation  of  the 
ionosphere  at  the  time  of  observation  [2],  The  method 
presented  reduces  errors  from  this  source  by  using 
information  about  the  ionosphere  implicit  in  the  received 
data  from  the  target  and,  if  available,  vertical  or  oblique 
sounders  and  other  HF  transmitters.  The  elevation  angles 
of  simultaneously  received  paths  from  any  given 
transmitter  inherently  contain  information  regarding  the 
real-time  ionosphere  conditions.  A  method  by  which 
electron  density  profiles  can  be  updated  from  arrival  angle 
measurements  of  known  transmitters  has  previously  been 
presented  [3],  The  updated  profiles  could  then  be  used  to 


improve  the  accuracy  of  SSL  determination.  The  present 
method  differs  in  that  a  joint  solution  is  made  for  both  an 
updated  ionosphere  model  and  the  distance  to  the 
unknown  transmitter  and  also  in  the  ease  with  which  data 
from  other  sensors  or  additional  parameters  to  be 
optimised  can  be  included  in  the  determination.  When  the 
input  data  just  consists  of  the  angles  of  arrival  of  1  and  2 
hop  paths  from  the  target  transmitter,  the  Nelder-Mead 
simplex  algorithm  is  used  to  minimize  a  multivariable 
function,  this  function  expressing  the  difference  between 
the  two  simultaneous  paths  as  a  function  of  the  various 
variables  of  the  real-time  ionosphere  that  are  optimised 
e.g.  foE,  foF2,  hbF2,  hmF2.  The  method  can  only  be 
applied  to  the  situation  where  two  simultaneous  paths  exist 
from  transmitter  to  receiver.  However,  this  is  a  condition 
which  is  often  satisfied  and  it  has  been  found  that  super¬ 
resolution  D/F  methods  can  be  used  to  successfully 
decompose  the  received  HF  wavefield,  received  on  a 
suitable  antenna  array,  into  its  constituent  component, 
giving  the  DOA  of  each  [4, 5, 6, 7, 8],  Thus  such  a  scheme 
compliments  more  traditional  SSL  methods  using 
interferometry  which  are  generally  only  performed  for 
single  moded  propagation. 

3.  ARRIVAL  ANGLE  DETERMINATION 

A  6  channel  HF  receiver  was  deployed  together  with  a  6 
element  2D  antenna  array  consisting  of  vertical 
monopoles.  Signals  received  by  the  array  were  input  to  the 
6  channel  receiver  and  each  mixed  down  to  20  kHz  before 
A/D  conversion  at  16  kHz..  Super-resolution  algorithms 
were  developed  to  efficiently  decompose  the  multi- 
component  HF  wavefield  into  its  constituent  paths 
[4,5,6,7,8].These  included  DOSE  (Direction  Of  arrival  by 
Signal  Elimination)  [4,5] ,  a  null-steering  algorithm,  which 
operates  on  the  data  vector  and  thus  can  work  on  single 
snapshot  data  and  fast  versions  of  MLE  which  operates 
either  in  element  [6,7]  or  beam  space  [8],  All  these 
algorithms  are  based  on  MLE  ensuring  that  they  are 
robust,  can  work  successfully  with  coherent  sources  (as 
arise  in  multipath)  and  also  can  also  operate  well  at  low 
SNR.  The  algorithms  were  found  to  very  significantly 
outperform  MUSIC  and  MVE  (CAPON)  both  on  real  and 
simulated  data  [4,9], 

The  azimuth  and  elevation  of  one  and  two  hop  paths  from 
the  unknown  transmitter  were  thus  obtained  for  input  to 
the  range-finding  program.  The  range-finding  program 
does  not  require  the  propagation  mode  to  be  identified  (e.g. 
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low/high  angle,  FI  or  F2  reflection);  only  the  number  of 
hops  (ionospheric  reflections).  The  ratio  of  the  tangents  of 
the  observed  elevation  angles  (tan  2-hop  angle/tan  1-hop 
angle)  can  be  determined  to  aid  this  investigation.  This 
value  will  be  2  for  a  flat  Earth  and  equal  virtual  reflection 
heights,  will  be  larger  than  2  for  greater  ranges  where  the 
curvature  of  the  Earth  is  important  and  can  be  less  than  2 
for  shorter  ranges  where  the  virtual  height  for  the  2  hop 
path  exceeds  that  of  the  1  hop  path.  The  array  was  not 
able  to  separate  o-  and  x-  modes  but  the  analytical  models 
used  to  model  the  ionosphere  in  the  optimisation  process 
are,  in  any  case,  unable  to  account  for  magnetic  field 
effects.  It  would  be  possible,  however,  to  develop  a  system 
which  considered  both  magneto-ionic  modes  if  an  antenna 
system  incorporating  crossed  loops  were  employed 
together  with  a  numerical  ray-trace  program  in  the 
function  minimisation  procedure. 

4.  RANGE-FINDING  PROGRAM 

The  program  is  operated  via  a  menu-driven  graphical 
interface  as  shown  in  figure  1.  This  is  first  used  to  input 
initial  data:  transmitter  frequency',  time  and  date  of 
measurement,  sunspot  number,  receiver  location  and 
angles  of  elevation  of  1  and  2  hop  paths  and  their  azimuth. 
The  sunspot  number  can  be  determined  from  the  date  if 
required.  The  method  first  determines  expected  parameters 
for  the  ionosphere  at  the  reflection  point(s).  The  simplest 
method  is  to  assume  that  the  ionospheric  parameters  are 
the  same  at  the  one  or  two  hop  reflection  points  as  then  it 
is  only  necessary  to  optimise  for  one  fixed  set  of 
ionospheric  parameters  and  the  ray  paths  can  be 
determined  from  analytic  formulae.  However,  it  is  also 
possible  to  assume  a  difference  in  the  ionosphere  between 
the  1  hop  and  2  hop  reflection  points  as  will  occur  if 
horizontal  gradients  are  present.  These  are  most  likely  to 
be  important  near  dawn  and  dusk.  In  this  case  a 
numerical  ray-tracing  program  or  a  segmented  MQP 
model  [10]  is  required  to  determine  the  ray  paths.  The 
entirety  of  both  1  and  2  hop  paths  are  then  determined 
rather  than  assuming  the  range  of  both  hops  of  the  2  hop 
path  are  identical.  In  this  case  optimisation  can  still  be 
performed  for  a  single  set  of  ionospheric  parameters  by 
introducing  the  difference  between  the  ionospheric 
parameters  at  the  different  reflection  points  by  way  of  a 
specified  model.  Alternatively,  the  optimisation  can  be 
performed  to  determine  ionospheric  gradients  as  well  as 
vertical  profiles  but  more  input  data  from  reference 
transmitters  (see  below)  would  be  required.  The 
optimisation  program  requires  as  good  as  possible  starting 
parameters  for  the  ionosphere  at  the  reflection  point(s). 
This  can  be  provided  either  just  on  the  basis  of  an 
ionospheric  model  or  using  real-time  information  from  a 
vertical  sounder  or  sounders  situated  at  the  location  of  the 
SSL  system  of  elsewhere  to  update  an  ionospheric  model. 
For  the  former  method,  ionospheric  parameters  are  first 
obtained  for  the  time  of  day,  date  and  receiver  location 
from  the  IRI90  (International  Reference  Ionosphere)  (step 
1).  Any  other  ionosphere  model  could  be  used  instead  if 


required.  An  estimated  distance  is  then  determined  using 
the  one  hop  elevation  angle  and  propagation  distance 
formulae  based  on  the  MQP  [11],  (step  2).  Using  the 
azimuth  angle  and  estimated  1-hop  distance,  a  new 
estimated  location  for  the  one  hop  reflection  point  is 
determined  (step  3).  This  new  location  is  then  used  to 
update  the  ionosphere  parameters  using  the  IRI90  model, 
(step  4).  The  steps  1  to  4  are  then  repeated  starting  with 
this  new  location.  An  additional  subroutine  is  used  to 
correct  the  initial  parameters  if  it  is  found  that  an  expected 
mode  cannot  be  reflected  by  the  ionosphere.  Finally  the 
IRI90  model  at  the  final  estimate  of  1  hop  reflection  point 
is  plotted  (see  figure  2).  If  real  time  ionospheric  profiles 
are  available  from  a  sounder,  this  data  can  be  used  to 
further  improve  the  estimate  of  the  electron  density  profile 
at  the  estimated  reflection  point(s).  An  analytical  model  is 
then  fitted  to  the  best  estimate  of  the  electron  density 
profile  at  the  reflection  point(s).  A  pull  down  menu 
(labeled  ‘model’)  can  then  be  used  to  plot  any  or  all  of  the 
ionospheric  profiles  based  on  the  three  models:  a  single 
quasi-parabolic  layer  [12],  a  modified  Bradley-Dudeney 
(CCDR)  ionospheric  profile  [13]  or  an  MQP  (multi-quasi- 
parabolic).  Greater  accuracy  in  fitting  the  MQP  model  to 
the  data  can  be  achieved  by  using  a  greater  number  of 
quasi-parabolic  layers  but  this  may  requires  a  careful 
choice  of  ionospheric  parameters  to  include  in  the 
optimisation  since  it  is  not  realistic  to  optimise  for  all  the 
parameters  of  all  the  layers.  The  fitted  profiles  are  plotted 
on  the  same  axes  so  that  their  correspondence  with  the 
estimated  electron  density  profile  at  the  reflection  point 
model  can  be  determined  (see  figure  2).  For  each  of  these  3 
analytical  models  (QP,  CCIR,  MQP), ,  analytical  formulae 
exist  whereby  the  propagation  distance  can  be  determined. 
Another  pull-down  menu  (labeled  ‘formula’)  can  then  be 
used  to  select  which  of  the  three  models  is  to  be  used  in  the 
optimisation  procedure.  In  all  these  analytical  models,  the 
effect  of  the  magnetic  field  is  neglected  so  that  individual 
magneto-ionic  modes  cannot  be  treated.  The  optimisation 
is  performed  to  determine  the  “best-fit”  ionospheric 
parameters  and  hence  propagation  distance.  Only  the  most 
important  parameters  of  the  model  are  treated  as  variables. 
These  can  be  determined  by  varying  each  parameter  in 
turn  over  a  given  range  e.g.  +  20%  corresponding  to  the 
likely  variation  of  the  parameter  in  the  real  ionosphere  and 
then  examining  the  effect  on  the  determined  propagation 
distance.  For  example,  in  the  case  of  the  modified  Bradley- 
Dudeney  model,  the  ionosphere  is  represented  using  two 
quasi-parabolic  layers,  representing  the  E  and  F2  layers 
and  a  quasi-linear  layer  in  between.  The  height  of  the 
maximum  and  semi-thickness  of  the  E-region  were  chosen 
as  fixed  parameters  with  values  of  1 10  and  20  km 
respectively.  Then,  there  are  4  variable  parameters,  foE, 
foF2,  hmF2,  ymF2;  the  optimisation  is  performed  for 
corrected  values  to  these  4.  The  fact  that  the  total  range  of 
the  2-hop  path  must  equal  that  of  the  1-hop  distance  is 
used  as  a  constraint.  The  algorithm  used  is  the  Nelder 
Mead  simplex  search  [1]  which  is  a  direct  search  method 
not  requiring  gradients  or  derivatives.  If  n  is  the  number  of 
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Figure  1  Input  Data  Screen 


Input  Data 
Button 


Figure  2  A  QP  profile  plotted  together  with  the  IRI90  profile 
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variables,  a  simplex  in  n -dimensional  space  is 
characterised  by  n+1  distinct  vectors  which  are  its  vertices 
e.g.  in  two-space,  a  simplex  is  a  triangle  and  in  three-space 
a  pyramid.  Using  the  appropriate  analytical  equations,  an 
analytical  function  can  be  determined  which  will  give  the 
propagation  distance  in  terms  of  the  chosen  variable 
ionospheric  parameters  of  the  chosen  ionospheric  model, 
the  elevation  angle  a  and  transmission  frequency  fin  the 
form,  say, 

D=F(foE,  foF2,  hmF2,  ymF2,  f,a). 

Then  the  following  function:  - 
|  F(foE,  foF2,  hmF2,  ymF2,  f,  a,)  -  2xF(foE,  foF2,  hmF2, 
ymF2,  f,  a2)  | 

will  have  a  minimum  for  the  distance  to  the  unknown 
transmitter  in  the  case  of  the  ionosphere  parameters 
assumed  the  same  at  both  the  1  and  2  hop  reflection 
points.  Thus  minimising  this  function  will  determine  an 
updated  “best  fit”  ionosphere  model.  If  the  ionosphere  is 
considered  to  vary  between  the  one  and  two  hop  reflection 
points,  then  the  following  function  is  minimised: - 
|  F(X)_,  X2,,  ....  Xn,,  f,  a,  ,P)  -F2(Xi,,  X2,,  ....  Xu,  f,  02  ,P)  I 
where  F)  ( )and  F2  ( )  are  functions  giving  the  1  and  2  hop 
ranges  respectively  in  terms  of  the  ionosphere 
parameters,  to  be  optimised  xu,  x2>, ....  x^ etc  which  may 
include  ionisation  gradients,  the  transmission  frequency  f, 
elevation  angles  a,,  and  azimuth  angle  p. 

There  is  also  a  choice  of  including  one  or  more  reference 
paths  to  known  transmitters  (see  section  5).  In  this  case, 
the  frequency,  propagation  distance  and  measured 
elevation  at  the  receiver  of  each  are  also  input  using  an 
additional  input  data  box  (see  figure  3).  Optimisation  is 
then  performed  to  find  the  same  parameters  of  the 
ionospheric  model  but  using  the  angle(s)  of  incidence  of 
the  unknown  target  transmitter  together  with  the  angles  of 
incidence  of  the  known  transmitters  for  which  the  known 
transmitter-receiver  distance  acts  as  an  additional 
constraint 

The  propagation  distance  can  finally  be  determined  from 
the  optimised  ionospheric  parameters  and  this,  together 
with  the  azimuthal  bearing  and  known  receiver  location, 
used  to  find  the  unknown  transmitter  location.  These  are 
then  printed  out  in  the  MiniCommand  Window  and  the 
ionospheric  profile,  based  on  the  chosen  analytical  model 
and  optimised  ionospheric  parameters,  also  plotted  (see 
figure  4). 

5.  USING  ADDITIONAL  INFORMATION 

It  is  clear  that  for  the  basic  method  the  number  of 
variables  optimised  exceeds  the  number  of  constraints  in 
the  system  so  that  w  here  all  four  initial  parameters  differ 
substantially  from  their  real  values,  good  convergence 
cannot  always  be  expected.  Additional  constraints  can  be 
included  in  the  system  by  (i)  including  information  on 
paths  to  known  transmitters  adjacent  to  that  to  the 
unknown  transmitter  (mentioned  above)  and/or  (ii)  using 


information  on  the  comparative  time  delay  of  the  different 
propagational  modes.  For  (i),  the  optimisation  is 
performed  to  find  the  same  parameters  of  the  ionospheric 
model  but  using  the  angle(s)  of  incidence  of  the  unknown 
transmitter  together  with  the  angles  of  incidence  of  the 

known  transmitters  of  frequency  fk  ,  for  which  the  known 
transmitter-receiver  distance  D,  acts  as  an  additional 
constraint.  Then,  for  example,  the  function  to  be 
minimised  could  be  of  the  form:- 

|  F(foE,  foF2,  hmF2,  ymF2,  f,  a,)  -  2xF(foE,  foF2,  hmF2, 
ymF2,  f,  a2)  |  +|  Z  F(foE,  foF2,  hmF2,  ymF2, 4,  ,aki)  -  D,  | 

where  the  summation  is  taken  over  all  /  known 
transmitters.  Different  weightings  could  be  applied  to  the 
first  term  and  each  of  the  /  terms  corresponding  to  known 
transmitters  if  some  measurements  were  considered  more 
accurate  than  others.  The  inclusion  of  this  extra 
information  can  significantly  improve  the  distance 
estimate.  More  successful  optimisation  can  be  achieved 
even  when  only  one  additional  reference  path  is  used.  This 
method  relies,  however,  on  the  ionosphere  being 
correlated  over  the  area  of  reflection  of  the  different 
transmissions.  Rush  and  Edwards  [14]  have  determined 
ionospheric  correlation  distances  in  both  North-South  and 
East-West  directions  for  different  latitude  ranges  and 
seasons.  For  mid-latitudes,  correlation  is  generally  very 
good  (coefficient  >  0.8)  for  points  separated  up  to  about 
1000  km  There  are  likely  to  be  many  known  transmitters 
within  this  range,  suggesting  the  practicality  of  this 
method.  The  correlation  measured  is,  however,  between 
hourly  deviations  from  the  monthly  mean  value  at  the 
respective  points.  Thus  a  more  accurate  method  would  be 
to  include  small  offsets  to  the  ionospheric  parameters  in 
the  equations  for  the  known  transmitters  so  that  for 
example,  foF2'  is  taken  as  foF2+s  where  fof2  is  the 
variable  being  optimised  and  s  is  the  difference  in  the 
ionospheric  model  between  foF2  at  the  midpoint  of  the 
path  to  the  known  transmitter  and  foF2  at  the  estimated 
midpoint  of  the  path  to  the  unknown  transmitter. 

6.  ACCURACY 

Although  ideally  the  method  should  be  tested  using  real 
data  consisting  of  experimentally  measured  angles  of 
arrival,  tests  were  first  made  using  modeled  data.  There 
were  three  reasons  for  this.  The  first  was  that  errors  in  the 
range-finding  method  itself  would  be  unambiguously 
determined.  For  measured  angles  of  arrival,  errors  in 
determined  distance  would  be  due  to  both  error  in  the 
measured  angles  of  elevation  as  well  as  any  errors  inherent 
in  the  method.  It  is  clearly  important  when  optimising  the 
method  to  achieve  greatest  accuracy  that  there  are  no 
errors  in  input  angles.  Secondly,  using  modeled  data,  the 
effect  of  random  errors  in  all  the  ionospheric  parameters 
e.g.  foE,  foF2,  hmF2  could  be  easily  determined.  Thirdly, 
it  was  considered  that  for  an  operational  system,  angles  of 
arrival  could  be  determined  more  accurately  than  using  the 
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Figure  3  Reference  Site  Information  Windows 
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Figure  4  Final  Screen  with  Results 
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veiy  limited  budget  equipment  built  by  ourselves  and 
employed  in  our  trials. 

Tests  against  modeled  data  were  accomplished  by  using  a 
random  number  generator  to  add  a  random  amount  (in  the 
range  ±  15%)  to  each  of  the  initial  values  of  the 
ionospheric  parameters  e.g.  foF2,  hmF2  that  were 
optimised  in  the  model.  The  amount  of  variation  was 
considered  commensurate  with  day  to  day  variation  in 
ionospheric  parameters  determined  by  [13,14],  Tests  were 
performed  for  different  path  combinations  such  as  1F2  and 
2F2.  It  was  found,  for  example,  for  a  560.5  km 
propagation  path  (see  fig.  5)  that  the  error  in  the 
determination  of  the  distance  to  the  target  transmitter 
using  the  MQP  model  was  less  than  2%  when  there  was 
one  added  reference  path  (which  was  1410  km). 
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Chirp  soundings  can  also  be  made  use  of  if  angles  of 
elevation  at  the  receiver  are  measured.  The  chirp  sounder 
can  be  considered  as  a  number  of  co-located  reference 
transmitters  of  differing  frequencies  of  which  each  one 
would  provide  an  additional  term  in  the  function  that  is 
minimised. 

Since  analytical  formula  exist  for  the  time  of  propagation 
as  well  as  the  distance,  the  measured  difference  in  time  of 
propagation  between  two  modes  can  be  used  as  an 
additional  constraint.  The  absolute  time  of  propagation  is 
not  required.  The  time  difference  can  be  measured  for  a 
modulated  transmitter  by  performing  an  autocorrelation  of 
the  received  baseband  signal.  The  accuracy  of  the 
measured  time  delay  will  of  course  depend  on  the 
bandwidth  of  the  modulation  but  can  be  improved  by 
employing  superresolution  techniques  in  the  time  domain. 
If  T=  F(foE,  foF2,  hmF2,  ymF2,  f,  a)  gives  the  time  of 
propagation  and  At  is  the  measured  difference  in 
propagation  time  between  say  1  and  2  hop  modes,  then  the 
following  function  should  be  minimised:- 

|  F(foE,  foF2,  hmF2,  ymF2,  f,  a,)  -  2xF(foE,  foF2,  hmF2, 
ymF2,  f,  a2)  |  + 1  T(foE,  foF2,  hmF2,  ymF2,  f,  a2)  -  T(foE, 
foF2,  hmF2,  ymF2,  f,  a,)  -  At] 

Different  weightings  could  also  be  applied  to  the  two  terms 
reflecting  different  accuracies  in  angle  and  time 
measurement. 


Figure  5  Distance  error  for  1F2  and  2F2  560.5  km  path 
with  an  additional  reference  transmitter  added. 

The  small  errors  shows  the  effectiveness  of  the  method 
although  more  trials  on  real  data  are  necessary.  The 
problem  with  trials  using  real  data  is  that  it  impossible  to 
separate  errors  in  determining  the  elevation  angles  using 
the  super-resolution  D/F  technique  with  errors  in  distance 
introduced  by  the  SSL  optimisation  process.  It  is 
considered  that  the  method  would  work  optimally  when 
elevation  angles  are  determined  to  about  0.  l°accuracy. 
Best  mode  separation  and  bearing  accuracy  can  be 
achieved  in  the  presence  of  multipath  when  employing  (i) 
an  array  long  compared  with  the  wavelength  of  the 
received  transmission  (ii)  a  3D  array  using  linear  active 
antenna  well  above  the  ground  and  (iii)  MLE  based  super- 
resolution  methods  [4--8]  which  can  best  cope  with  the 
coherent  sources  that  arise  in  multipath.. 

7.  MAGNETO-IONIC  MODES  AND  TILTS 

The  optimisation  can  be  performed  for  a  more  realistic  3D 
ionosphere  model  and  including  the  effect  of  the 
geomagnetic  field  if  numerical  ray-tracing  rather  than 
analytical  formulae  are  used  in  the  optimisation  process. 
This  would  enable  the  two  magneto-ionic  modes  to  be 
treated.  For  a  3D  ionosphere  model,  optimisation  could 
also  be  performed  for  additional  parameters  such  as  a 
horizontal  gradient  of  electron  density.  The  use  of  a 
numerical  ray-tracing  method  does,  however,  greatly 
increase  the  time  required  for  the  solution. 


8.  CONCLUSIONS 

A  new  SSL  method  for  HF  transmitters  has  been  described 
which  employs  a  Nelder  Mead  minimisation  of  a  multi¬ 
variable  function  which  can  include  data  from  a  variety  of 
sources.  Thus,  the  final  determination  of  the  location  of 
the  target  transmitter  can  be  based  on  data  from  the 
following  sensors:- 

1.  Multi-element  receiving  antenna  array 

2.  Vertical  and/or  oblique  ionospheric  sounders 

3.  Passive  reception  of  known  HF  transmitters 

4.  Chirp  sounders 

5.  Relative  time  of  arrival  of  multimode  components 
An  optimum  solution  for  both  an  updated  real-time 
ionosphere  model  as  well  as  target  transmitter  location  is 
obtained  which  is  a  single  solution  employing  all  the 
measured  data. 

The  SSL  determination  is  performed  under  MATLAB  via 
a  menu-driven  graphical  user  interface.  This  employs  pull¬ 
down  menus  for  data  input,  program  choice,  ionospheric 
profile  plotting  and  presentation  of  results. 
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PAPER  No:  22 

DISCUSSOR’S  NAME:  P.  Cannon 

COMMENT/QUESTION: 

You  have  used  MQP  and  have  shown  that  your  errors  are  small.  MQP  does  not  allow  for  spatial 
variations  along  the  path.  Could  you  please  comment? 


AUTHOR/PRESENTER’S  REPLY: 

The  figures  presented  were  tto  show  errors  due  to  departure  of  real-time  ionosphere  parameters 
from  parameters  determined  from  an  ionosphere  model  with  up  to  15%  difference  determined 
using  a  random  number  generator  independently  for  each  parameter.  These  calculations  are  for 
an  ionosphere  with  no  horizontal  gradients.  In  the  written  paper  I  mention  how  horizontal 
gradients  could  also  be  included  but  requires  either  numerical  ray-tracing  using  e.g.  Jones  30 
program  or  the  segmented  MQP  mdoel  which  my  questioner  has  devised  with  R.J.  Norman  (my 
reference  10). 


DISCUSSOR’S  NAME:  C.  Goutelard 

COMMENT/QUESTION: 

When  you  introduce,  as  you  indicate,  the  horizontal  components  of  the  ionisation  gradients,  are 
you  sure  that  you  will  always  obtain  a  unique  solution  for  your  optimisation. 


AUTHOR/PRESENTER’S  REPLY: 

The  simulations  presented  in  my  talk  were  for  an  ionosphere  with  no  horizontal  gradients. 
However,  the  method  can  be  applied  for  such  an  ionosphere  (as  detailed  in  the  written  paper)  by 
employing,  either  a  ray-tracing  program  utilising  numerical  integration  (e.g.  Jones  30)  or  a  spatially 
segmented  MQP  (see  my  reference  10),  in  each  determination  of  the  function  that  is  minimised.  I 
plan  to  do  further  trials  to  investigate  this.  In  general  the  problem  of  a  non-unique  solution  can  be 
circumvented  by  making  sure  that  the  initial  values  chosen  for  parameters  in  the  optimisation  are 
close  to  their  real  values. 
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PAPER  No.  22 


DISCUSSOR’S  NAME:  G.  S.  Brown 


COMMENT/QUESTION: 

The  results  of  your  simulation  seem  to  contain  a  bias  in  them;  have  you  run  enough  simulations  to 
see  what  the  histogram  of  the  error  looks  like? 

AUTHOR/PRESENTER’S  REPLY: 

The  number  of  sets  of  random  errors  presented  in  each  figure  was  32.  I  think  that  this  is  too  small 
a  number  to  be  sure  that  the  overall  percentage  error  in  distance  would  be  expected  to  be  zero. 
However,  I  intend  to  investigate  the  possible  existence  of  any  systematic  errors  by  conducting 
similar  trials  with  larger  numbers  of  data  sets  including  random  errors. 
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ABSTRACT 

This  paper  examines  the  problem  of  incorporating  smart 
antennas  in  wireless  DS-CDMA  systems.  The  use  of  ar¬ 
ray  signal  processing  in  DS-CDMA  systems  offers  the  pos¬ 
sibility  of  jointly  utilizing  code  and  spatial  resources  to 
achieve  superior  performance  compared  to  conventional 
multiple-access  methods.  The  development  of  interference 
suppression  techniques  is  essential  to  take  full  advantage  of 
the  available  radio  resources  and  requires  data  fusion  in  the 
spatial  and  code  domains.  Spatial  signal  processing  can  also 
lead  to  faster  and  robust  synchronization  and  provide  the 
capability  to  rapidly  adapt  to  channel  fading,  co-channel 
and  adjacent  channel  interferers.  The  overall  method  is 
suitable  for  real-time  implementation  and  can  lead  to  a  di¬ 
rect  increase  in  the  overall  capacity,  quality  and  coverage  of 
these  systems. 

1.  INTRODUCTION 

The  use  of  spread  spectrum  multiple  access  techniques  is 
well  established  in  the  wireless  communications  arena.  In 
particular,  Direct  Sequence  Code  Division  Multiple  Access 
(DS-CDMA)  has  been  widely  studied  in  the  literature  and 
has  been  implemented  in  several  commercial  systems  as 
well.  Adaptive  interference  suppression  techniques  based 
on  multiuser  detection  and  antenna  array  processing  have 
recently  been  considered  as  powerful  methods  for  increasing 
the  quality,  capacity  and  coverage  of  these  systems.  They 
provide  a  superior,  though  computationally  more  expensive, 
alternative  to  conventional  single  sensor  (matched  filtered) 
detection  which  is  severely  limited  by  multi-access  interfer¬ 
ence. 

This  paper  focuses  on  co-channel  interference  (CCI)  mit¬ 
igation  techniques  for  the  uplink  channel  of  DS-CDMA  sys¬ 
tems  over  frequency  selective  slowly  fading  channels.  Mul¬ 
tiuser  detection  [11,  16,  24]  and  adaptive  array  processing 
[2,  18,  22,  25]  have  been  shown  to  be  promising  solutions 
to  this  problem.  By  carrying  out  multi-sensor  data  fusion 
using  an  antenna  array,  significant  CCI  reduction  can  be  ob¬ 
tained.  Furthermore,  this  approach  is  particularly  effective 
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when  data  fusion  is  done  adaptively  to  combat  time- varying 
channels  and  interference.  Towards  this  end,  we  present 
here  a  novel  adaptive  array  receiver  structure  which  adopts 
a  two-pronged  approach  to  CCI  suppression  using  multiuser 
detection  and  beamforming.  Furthermore,  a  RAKE  struc¬ 
ture  is  retained  for  overcoming  frequency  selective  fading. 
Thus,  the  overall  receiver  makes  a  combined  use  of  fre¬ 
quency  diversity  (through  the  use  of  RAKE  combining), 
code  (or  time)  diversity  and  angle  diversity  (through  the 
use  of  beamforming). 

The  principles  of  multiuser  detection  are  by  now  well 
established  [11,  24],  The  optimal  detector  [24]  is  a  ma¬ 
jor  theoretical  milestone  but  has  limited  practical  utility 
due  to  its  exponential  complexity  in  the  number  of  users. 
Decorrelating  detectors  have  enjoyed  much  popularity  due 
to  their  near-far  resistant  performance  and  much  reduced 
linear  complexity.  A  minimum  energy  formulation  for  blind 
adaptive  near-far  resistant  multiuser  detection  was  intro¬ 
duced  in  [11]  and  also  used  in  [22].  While  the  MMSE  (and 
the  equivalent  minimum  energy)  solution  is  known  to  be 
near-far  resistant,  a  major  obstacle  lies  in  adaptively  seek¬ 
ing  the  optimum  solutions.  Stochastic  gradient  algorithms 
such  as  the  Least  Mean  Squares  (LMS)  method  are  very  at¬ 
tractive  because  of  their  simplicity  but  suffer  from  poor  con¬ 
vergence  due  to  the  large  dimensionality  in  typical  CDMA 
systems  and  the  fluctuation  in  received  signal  powers  [16]. 
Least  squares  (LS)  algorithms  on  the  other  hand,  provide 
adequate  convergence  speed  under  these  circumstances,  but 
can  be  prohibitively  complex  to  implement.  Note  that  most 
low  complexity  fast-RLS  type  algorithms  are  not  applicable 
due  to  the  lack  of  time-shift  structure  in  the  input  vectors 
[10].  Thus,  there  is  a  clear  imperative  to  develop  reduced 
complexity  multiuser  detection  schemes  without  having  to 
sacrifice  performance.  The  major  difference  in  the  adap¬ 
tive  multiuser  detector  presented  in  this  paper  is  that  it 
utilizes  a  novel  recursive  update  algorithm  based  on  set- 
membership  parameter  estimation  theory  which  provides 
significant  benefits  in  terms  of  tracking,  convergence,  and 
complexity. 

Adaptive  arrays  are  typically  designed  for  either  utiliz¬ 
ing  spatial  diversity  for  mitigating  the  effect  of  fading  or 
for  using  the  inherent  angle  diversity  in  the  received  sig¬ 
nals  for  CCI  reduction.  In  [15],  an  adaptive  array  receiver 
configured  as  the  cascade  of  a  beamformer  and  matched  fil¬ 
ter  detector  is  presented.  The  update  of  the  beamformer 
weights  in  this  structure  is  rather  cumbersome,  especially 
over  fading  channels.  A  receiver  structure  with  a  matched 
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filter  detector  followed  by  a  beamformer  is  presented  in 
[18].  However,  the  DOA  estimation  algorithm  used  for 
beamforming  is  computationally  complex  (it  is  based  on 
eigen-decompositions)  and  is  sensitive  to  angle  spread.  An 
extension  of  this  receiver  structure,  called  a  2D-RAKE  is 
also  presented.  The  2D-RAKE  is  designed  to  operate  over 
multipath  channels  and  consists  of  a  front-end  beamformer 
which  feeds  the  beam-steered  signals  into  a  standard  RAKE 
receiver  structure.  Joint  spatio-temporal  reception,  akin 
to  broadband  beamforming  [23],  has  also  been  considered 
[2,  16]. 

2.  CDMA  SIGNAL  MODEL  AND  PROBLEM 
FORMULATION 

Consider  an  asynchronous  DS-CDMA  system  with  K  users. 
Such  a  system  may  be  used  for  a  cellular  wireless  network 
with  the  K  users  distributed  in  a  cell  of  interest.  The  cen¬ 
tralized  receiver  at  a  cellular  base  station  receives  asyn¬ 
chronous  transmissions  from  all  active  users  on  the  uplink 
(mobile  to  base)  multiple  access  channel  and  interfering  sig¬ 
nals  from  adjacent  cells  as  well.  Let  the  information  sym¬ 
bol  sequence  from  the  kth  user  be  denoted  by  Ak(-),  chosen 
from  a  possibly  complex  alphabet.  Assuming  the  symbol 
and  chip  duration  to  be  Ts  and  Tc,  respectively,  resulting  in 
a  nominal  processing  gain  N  -  T,/Tc,  the  spread  spectrum 
signal  emanating  from  the  kth  user  is  given  by 

Mt)  =  Y  ^EAk(l^\)ck(n%N)5(t  -  nTc)  (1) 

n 

where  n  is  the  chip  index;  %  denotes  the  modulus  operation 
and  [J  denotes  the  floor  operation;  Pk  is  the  transmitted 
power  of  the  kth  user;  ck{n%N)  denotes  the  nth  chip  of  the 
kth  users’  periodic  spread  spectrum  sequence  with  period 
N .  After  performing  baseband  pulse  shaping  with  a  filter 
ip(t),  the  transmitted  waveform  is  given  by 

xk(t)  —  Y,  VPkAk([~\)ck(n%N)ip(t  -  nTc)  (2) 

n 

The  pulse  t/>(f)  is  assumed  to  have  unit  energy  and  duration 
Tc.  Assume  that  the  transmitted  signal  from  the  fetli  user  is 
received  at  an  antenna  array  receiver  with  M  elements.  A 
L-path  frequency  selective  slowly  fading  model  is  assumed 
in  order  to  formulate  the  spatio-temporal  impulse  response 
of  the  mth  sensor  to  the  signals  from  the  kth  user  (see 
Appendix  A)  according  to 

him)(t)  =  Ya[^5(t-rklTc)  (3) 

2=0 

where  denotes  the  composite  response  of  the  mth  sen¬ 
sor  to  the  Zth  multipath  component  from  the  kth  user;  rkiTc 
is  the  time  delay  of  the  Zth  multipath  component  of  the 
kth  user’s  received  signal  and  that  the  maximum  delay  is 
Tk(L-i)Tc  <  Ts.  Thus,  the  received  complex  baseband  sig¬ 
nal  from  the  kth  user  at  the  mth  element  is  given  by  the 
convolution 

ri m)(0  =  *k(t)*h[nl)(t)  =  Ya<U)Zk(t-TkiTc) 

1=0 


=  Y^  VPkAkdjj \)ck(nVoN) 

n 

L  —  l 

Yak f^t-nTc-mTc)  (4) 

1=0 

The  received  signal  from  K  asynchronous  users  at  the  cen¬ 
tralized  receiver  is  obtained  as  the  superposition  of  each 
user’s  signals  according  to 

K—l 

r(^(t)  =  Yrim)(t-uk)  +  n(m\t)  (5) 

fc= o 

where  vk  is  the  flat  propagation  delay  of  the  kth  user  and 
«*"**(<)  denotes  the  mth  sensor’s  front-end  additive  noise. 
The  receiver  structure  used  in  Figure  8  is  used  to  recover 
transmitted  symbols  from  all  desired  users.  For  a  particular 
desired  user,  the  composite  signal  is  chip  match  filtered  and 
fed  into  multiple  RAKE  arms,  each  delayed  by  a  chip  time 
or  more.  It  is  assumed  that  the  receiver  uses  conventional 
techniques  for  determining  which  RAKE  arms  contain  de¬ 
layed  copies  of  the  signal  [18].  Also,  RAKE  receiver  arms  for 
user  k  are  synchronized  to  the  path  delays  rkiTc.  Without 
loss  of  generality,  chip  synchronous  reception  is  assumed. 
Thus,  the  chip  rate  samples  for  the  kth  user  with  delay 
tuTc  are  obtained  at  the  mth  sensor  after  demodulation  as 

-  mTc  -  I/*))  |t=nr,  (6) 

Denoting  the  sequence  of  received  samples  of  dimension  N 
spanning  one  symbol  as  v^"!,  the  chip  sample  vector  for  the 
pth  symbol  of  the  desired  user,  i.e.,  for  n  €  \pN,pN+N— 1], 
is  given  by 

vk T\p)  =  '/Pka^)  Ak{p)ck 

+  ^  '  '/PrO,\.r  '[Afc.oCfc^o  +  Afc^iCfcr^-i] 

r^l 


L—l 


h^k  2=0 


[Ah,oChi,o  +  Ah,-iChi,-i]  +  n(m)(p)  (7) 

where 

C*  =  [ci(0),Ck(l),  •  ■  ■  ,ck(N  —  1)]T 

Cfci.o  =  [0,  ...,0,  cOO),...,  ck(N  -n,Tc  -  1)]T 

=  [ck(N  -Tk,Tc),...,ck{N  -  1),0,...,0]T(8) 

and  Aa,,o  and  Ak^-\  denote  the  two  overlapping  symbols; 
n(m)(p)  is  the  vector  of  filtered  noise  samples  during  the 
pth  symbol.  Thus,  in  addition  to  the  filtered  background 
noise,  each  sample  of  the  received  chip  vector  for  the  pth 
symbol  has  two  interfering  components.  These  arise  from 
the  multipath  components  of  the  same  user  with  different 
time  delays  and  from  all  other  users.  In  addition,  each  inter- 
ferer  (self-multipath  as  well  as  from  other  users)  contributes 
two  independent  interference  vectors  to  the  received  sam¬ 
ple  vector  in  each  symbol  time.  The  kth  users’  symbols  are 
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extracted  using  a  linear  detector  wi: ,  characterized  by  the 
discrete  time  inner  product  with  the  sampled  chip  sequence 
as, 

Ai\p)  =  wfvlrV)  (9) 

As  can  be  seen  in  Figure  8,  a  linear  multiuser  detector  is 
used  for  each  RAKE  branch  at  all  array  elements.  The 
computation  and  adaptive  update  of  these  linear  detectors 
is  now  addressed. 

3.  BLIND  MULTIUSER  DETECTION 

The  goal  here  is  to  blindly  compute  linear  decorrelation 
weight  vectors  to  preserve  the  desired  signal  and  mitigate 
interference,  see  e.g.,  [11,  22]  for  prior  work.  For  the  spe¬ 
cial  case  that  the  receiver  treats  multiple  access  interfer¬ 
ence  as  additive  white  noise  (or  when  detector  adaptivity 
is  not  feasible)  the  linear  detector  is  merely  the  signature 
sequence  of  the  desired  user.  The  following  algorithm  is 
presented  following  the  framework  of  the  well  known  Gen¬ 
eralized,  Sidelobe  Canceler  (GSC)  [23],  Constraining  the 
detector  to  present  a  unit  response  to  the  desired  user’s 
signature  sequence,  and  decomposing  wi  into  correspond¬ 
ing  constrained  and  unconstrained  components  denoted  by 
Wk,q  and  „  respectively,  we  have 

w k  =  W k,q  -  Cfc,nW k,a  (10) 

whprp 

Wfc,,  =  ct(c?c*rIs  (ii) 

with  the  constraint  matrix  C*,  =  c*,  and  the  output  con¬ 
straint  p=l.  Thus,  in  this  case,  w =  Ck/N.  The 
columns  of  the  N  x  {N  —  1)  matrix  C/t,„  span  the  null  space 
of  Ck .  In  general,  of  course,  multiple  constraints  can  be 
imposed  on  w ii<7,  for  instance,  to  exploit  the  knowledge  of 
signature  sequences  and  timing  of  other  interfering  users. 
Such  constraints  appear  as  columns  of  C*  and  result  in  a 
corresponding  decrease  in  the  dimensionality  of  Wfc,a.  In  the 
extreme  case  when  all  the  columns  of  C k  are  constrained, 
the  detector  ceases  to  be  adaptive.  Ck,n  is  readily  obtained 
via  one  of  many  orthogonalizing  procedures  [23].  In  fact, 
for  the  special  case  above,  Ck,n  can  be  precomputed  off¬ 
line  for  each  desired  signature  sequence.  The  unconstrained 
component  w k,„  represents  the  N  —  1  dimensional  adaptive 
portion  of  wfc.  Thus,  the  output  of  the  mth  array  element 
for  the  Zth  symbol  is  given  by 

4 7}(p)  =  W ",vi7)(p)  -  "nvi?)(p)  (12) 

Equation  (12)  can  be  viewed  as  a  standard  adaptive  filter¬ 
ing  problem  with  w**  vjj^(p)  serving  as  the  desired  signal; 
C k,n^<ki‘) (p)  serving  as  the  input  vector;  \Vk,a  denoting  the 
adaptive  weight  vector  and  el™  *  ( p )  denoting  the  estimation 
error, 

3.1.  Adaptive  Solution  Strategies 
The  conventional  solution  to  the  above  is  to  perform  an 
unconstrained  optimization  involving  wk,a  which  can  be 
obtained  via  adaptive  algorithms  such  as  stochastic  gra¬ 
dient  descent  or  least  squares  [10].  A  compelling  solution 
to  this  problem  is  to  consider  a  new  approach  based  on 
set-membership  parameter  estimation  theory.  For  excellent 


tutorial  overviews  of  set-membership  theory,  see  [3]  and  [6]. 
Other  key  papers  detailing  the  structure,  features,  conver¬ 
gence  and  tracking  properties,  and  signal  processing  appli¬ 
cations  of  set-membership  algorithms  include  [4,  5,  7]  and 
the  references  therein. 

Specifically,  the  focus  of  this  paper  is  on  a  subset  of  set- 
membership  techniques,  namely  the  class  of  optimal  bound¬ 
ing  ellipsoids  (OBE)  algorithms.  There  are  several  fea¬ 
tures  of  OBE  algorithms  which  render  them  attractive  for 
the  problem  at  hand.  Experience  has  shown  that  OBE 
algorithms  perform  better  than  weighted  recursive  LS  al¬ 
gorithms  in  tracking  time  varying  parameters  and  in  low 
signal-to-noise  ratio  (SNR)  situations  [4,  8,  13].  Further¬ 
more,  OBE  algorithms  are  computationally  efficient  due  to 
their  discerning  update  property.  They  can  also  provide  an 
explicit  indication  of  any  loss  in  tracking  -  a  feature  not 
possessed  by  point  estimation  algorithms.  Although  LS  al¬ 
gorithms  can  also  be  equipped  with  such  indicator  functions 
[10],  such  a  feature  is  an  integral  part  of  parameter  estima¬ 
tion  using  OBE  algorithms.  As  a  point  of  common  ground, 
the  geometric  centers  of  the  bounding  ellipsoids  in  OBE 
algorithms  (which  are  usually  taken  as  point  estimates  at 
any  given  time)  are  known  to  be  weighted  recursive  LS  esti¬ 
mates  [6].  Simply  stated,  the  optimization  of  the  weighting 
(update)  factors  of  data  sets  according  to  set-membership 
principles  essentially  leads  to  the  discerning  update  prop¬ 
erty  and  superior  convergence  and  tracking  properties.  The 
OBE  algorithms  may  appear  to  have  0(N2)  complexity 
from  an  inspection  of  the  recursive  update  equations.  How¬ 
ever,  their  discerning  update  (or  data  selective)  feature  can 
be  fruitfully  exploited  for  significant  reduction  in  complex¬ 
ity  [5,  8].  In  particular,  O(N)  implementations  are  de¬ 
scribed  in  [5,  9]  and  an  efficient  time-shared  implementation 
between  multiple  users  is  described  in  [8].  Thus,  the  struc¬ 
ture  and  properties  of  OBE  algorithms  can  narrow  the  gap 
between  performance  and  complexity  which  is  encountered 
by  several  conventional  adaptive  filtering  algorithms. 

3.2.  An  OBE  Algorithm 

A  recursive  algorithm  is  now  derived  to  estimate  w k,a  based 
on  set-membership  principles  using  OBE.  The  idea  here  is 
to  update  the  estimator  such  that  the  estimation  error  is 
constrained  to  lie  within  a  specified  performance  bounds. 
The  goal  is  to  construct  an  OBE  algorithm  which  attempts 
to  seek  solution  vectors  wk,a  that  meet  the  following  speci¬ 
fication  for  all  p, 

M- 1 

E  \<A7](p)  -  w"„C^nv'7)(p)|2  <  7 2  (13) 

m=0 

where  qf  is  a  specified  constant  corresponding  to  a  desired 
performance  level  and  may  also  be  viewed  as  a  design  pa¬ 
rameter.  In  this  paper,  a  suitably  selected  constant  7 1  is 
assumed  (see  [6,  7]  for  general  guidelines  on  selection  of  7^). 
Weight  vectors  which  achieve  (13)  for  all  possible  input  se¬ 
quences  constitute  the  so  called  feasible  set.  The  objective 
of  the  OBE  methodology  is  to  seek  this  feasible  set  by  suc¬ 
cessively  refining  membership  sets  W4(p)  defined  at  time  p 
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as 


V  M  —  1 

m(P)  =  f|<w  e  cN~l  ■  £  Iwm^,(p) 

r =0  m=0 

-wflC^W(rf<7l}  (14) 

However,  the  complexity  in  exactly  computing  V\4(p)  is 
overwhelming  even  for  small  N  and  p.  OBE  algorithms 
circumvent  this  problem  by  recursively  updating  hyper¬ 
ellipsoids  £k(p)  which  tightly  outerbound  the  membership 
sets  Wkip)  for  all  p.  It  follows  that  £k  (p)  also  outerbounds 
the  feasible  set  at  all  times,  since  the  feasible  set  is  a  subset 
of  V\4(p)  for  all  p.  Thus,  a  recursive  formulation  can  be 
used  to  update  £k{p)  with  each  incoming  chip  vector  at  the 
symbol  rate.  Equation  (12)  can  be  rewritten  in  matrix  form 
for  the  arraj'  as, 

eI/(p)  =  Ufc,,(p)  -  V"„(p)wfc,a  (15) 

where 


vM(p) 

=  [vu)(p),VH)(p),--vif  1}(P)] 

U  m(p) 

=  V",(p)wi;,9  and 

■  Vfc,„(p) 

=  [C^v^^.C^v'^p), 

•••>CLvif_1)(p)] 

(16) 

Then  as  per  the  set-membership  framework,  let  the 

error 

specification  for  the  kth  user  be, 

l|e*((p)H2  <  ll  (17) 

where  ||  •  ||  denotes  the  vector  h  norm  and  q2  is  an  appro¬ 
priately  chosen  constant.  The  selection  of  7 £  is  addressed 
later  in  this  section.  Define  Sk{p )  as  a  degenerate  ellipsoid 
as 

Sk(p)  =  {w  e  CN~l  :  ||Ufc,q(p)  -  v£„(p)w||2  <  7!}  (18) 

Let  the  membership  set  at  time  (p  —  1)  be  given  by 

£k(p~  1)  =  {weC*"1  :[w-w,(p-l)]"p-1(p-l) 

[w-  wfc(p-  1)]  <  ol(p-  1)}  (19) 


is  a  well  defined  ellipsoid. 

Proposition  3.1.  Consider  the  inequalities  (18)  and  (19) 
above.  Define 

d  k(p)  =  UM(p)-Vfn(p)wfc(p-l) 

G  k(p)  =  v£n(p)Pfc(p-l)V*,n(p) 

Q k(p)  =  [1  -  Ajt(p)]lM  +  Xk(p)Gk{p) 


The  following  recursive  update  equations  may  be  obtained, 

Pi(p)  =  r^^y[Pt(p_1)-Afc(p) 

pfc(p-  i)vfc,4p)Q^(p)v£„(p)p*(p- 1)], 
Wfc(p)  =  Wfc(p  -  1)  +  Afc(p)Pfc(p)Vfc,„(p)dfc(p)  and 
°l{p)  =  [1  —  Xk(p)]crl(p  —  1)  +  Xk(p)yl 

“I1  -  Afc(p)]Afc(p)df  (p)Qi '(p)dt(p)  | 

The  last  three  equations  in  Proposition  3.1  constitute  the 
recursions  of  the  OBE  algorithm.  In  order  to  compute  the 
optimal  update  factor  the  parameter  cr£(p)  is  minimized. 
The  parameter  cr2(p)  can  be  considered  as  a  bound  on  the 
estimation  error  at  the  pth  step  and  is  closely  related  to 
other  popular  measures  of  optimization  such  as  volume  and 
trace  of  the  bounding  ellipsoid  [4,  13].  Unlike  these  mea¬ 
sures,  minimization  of  <r£(p)  lends  itself  to  a  very  efficient 
test  for  innovation.  A  tight  upper  bound  on  cr£(p),  denoted 
by  cr'2(p),  is  given  by 


a'2(p)  =  [1  -  A*(p)K2  (p  —  1)  +  A k(phl  -  [1  -  A*(p)] 

df  (p)d*,(p) 


Afc(p) 


[1  -  Afc(p)]  +  A k(p)gk(p) 


(22) 


where  gk {p)  =  1 1 Gk (p) 1 1 .  Denote  the  optimal  A*, (p)  by  A °k{p) 
(which  lies  in  [0,  tu]  for  some  ak  <  1)  and  define  the  quan¬ 
tity 


d"(p)d*(p) 


(23) 


Proposition  3.2.  Minimization  of  <r'2(p)  with  respect  to 
Afc(p),  leads  to  the  following  update  condition: 


(!)  if  pk{p)  >  1,  then  A °k{p)  =  0 


where  Pk(p-  1)  is  a  symmetric  positive-definite  matrix  and 
wj t(p  -  1)  is  the  center  of  the  ellipsoid.  An  ellipsoid  that 
contains  £k(p  —  1)  fl  Skip)  is  given  by 

Skip)  =  {w  €  C"_1  :  [1  -  Afc(p)] 

(w-w^-lfp-^-l) 

[w  -  wk{p  -  1)]  +  A*(p)||Ufc,,(p)  -  V*nw||2 
<  [1  -  A*(p)]cr2(p-  1)  +Afc(p)7f}  (20) 

where  A kip)  is  a  real  number  in  [0, 1].  It  can  now  be  shown 
that  there  exists  a  symmetric  positive  definite  P  k{p)  and  a 
positive  scalar  <r2(p)  such  that 

£k(p)  =  {wee""1  :[w-w*(p)]"P^(p) 

[w  -  w*(p)]  <  <r2(p)}  (21) 


(2)  otherwise  A °k(p)  —  min(ajt, vk(p))  where 


Vk{p) 


ak  if  df  (p)dfc(p)  =  0; 


(1  -/?fc(p)) 
2 


if  9k{p)  =  1; 


1  /  9k(p)  _ , 

1~9k(p)i  V  1  +  Pk(p)[gk(p)  -  1]  ’ 

if  1  +  0k{p)[gk{p)  —  1]  >0; 

as,  if  1  +  f3k{p)[gk(p)  -  1]  <  0  |  (24) 


See  [13]  for  proof  of  Propositions  3.1  and  3.2.  This  result  is 
used  for  computing  the  optimal  update  parameter  A Up)  at 
each  step.  Equation  (23)  and  Proposition  3.2  constitute  the 
condition  for  data  selectivity  or  no-update.  At  any  time  p, 
the  center  of  the  ellipsoid,  w k{p)  is  taken  as  an  estimate  of 
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the  adaptive  component  of  the  detector  in  (10).  To  initialize  to  p,  the  MLE  of  the  spatial  phase  difference  can  be  ob- 
the  algorithm,  £/t(0)  is  chosen  such  that  tained  straightforwardly  (see  Appendix  B)  as 


£fc(0)  =  (w  eCN  1  :  ||w||2  <  1/p) 

where  p  -C  1  resulting  in  £*(0)  being  a  suitably  large  initial 
hyper-sphere.  In  other  words,  the  following  initial  values 
may  be  chosen 


P*:(0)  =  Tv-1,  wjt(O)  =  0  and  crjfe(O)  =  1/p  (25) 

The  next  stage  of  processing  in  the  receiver  of  Figure  8 
entails  the  combining  of  the  detector  outputs  corresponding 
to  each  RAKE  branch  from  all  array  elements  via  a  bank 
of  beamformers. 

4.  DOA  ESTIMATION  AND  BEAMFORMING 

Carrying  out  DOA  estimation  prior  to  despreading  can  be  a 
formidable  task.  Most  subspace  based  methods  are  not  ap¬ 
plicable  due  to  the  large  number  of  independent  signals  and 
typically  small  number  of  array  elements.  Post-detection 
DOA  estimation,  on  the  other  hand,  is  a  viable  option.  Sev¬ 
eral  methods  have  been  proposed  in  the  literature,  includ¬ 
ing  subspace  based  methods  such  as  Weighted  Subspace 
Fitting  (WSF),  ESPRIT  [20]  and  an  eigendecomposition 
based  method  [18].  The  structure  of  a  DS-CDMA  system, 
however,  allows  for  a  much  simpler  and  robust  approach. 
This  section  describes  an  approximate  Maximum  Likelihood 
DOA  Estimator  (DOA-MLE)  used  for  beamforming.  In  ad¬ 
dition  to  inheriting  the  desirable  properties  of  MLEs,  it  also 
turns  out  to  be  simple  and  intuitive.  The  basic  idea  is  to 
partition  the  array  into  groups  of  two  consecutive  sensors 
or  doublets.  The  algorithm  then  exploits  the  fading  corre¬ 
lation  between  closely  spaced  doublet  elements  to  extract 
the  spatially  induced  phase  differences  between  the  post- 
correlation  complex  baseband  outputs.  Each  doublet  oper¬ 
ates  independently  to  compute  the  DOA-MLE  and  the  es¬ 
timates  from  multiple  doublets  are  then  suitably  combined. 


im  [EU-f+14?+1)W(47)W),]~ 

(27) 

where  (ej^M)*  denotes  the  complex  conjugate  of  e\ 

At  every  time  instant,  each  doublet  contributes  a  DOA- 
MLE  for  each  user  being  tracked.  Since  all  array  elements 
are  used  for  beamforming,  the  DOA-MLEs  from  each  dou¬ 
blet  are  combined  to  form  a  single  estimate.  The  simplest 
method  is  to  average  the  outputs  from  each  doublet  and 
other  schemes  may  be  readily  conceived.  For  instance,  a 
suitable  selection  criterion  can  be  adopted  for  selecting  the 
“best”  DOA-MLE  from  among  the  doublets.  In  any  case, 
denote  the  final  DOA  estimate  at  time  p  by  4>ki{p)-  Let 
the  Ith  beamformer  weight  vector  be  denoted  by  bu  (see 
Figure  8);  the  data  covariance  matrix  at  the  output  of  the 
Zth  RAKE  branch  by  R;  and  the  steering  vector  for  the  kth 
users’  Zth  path  be  given  by 

sfci(p)  =  (28) 


<^l7)  (p)  =  tan  1 


Adopting  the  classical  minimum  variance  distortionless  re¬ 
sponse  (MVDR)  criterion  for  computing  the  beamformer 
weights  [23] 


bfci(p)  = 


R;  1Skl(p) 

Skl(p)RTlskl(p) 


(29) 


The  output  of  the  Zth  beamformer  is  then  given  by 


Vki(p)  =  bfi  (p)eu(p)  (30) 


bfcj(p)  is  typically  block  updated  using  (29)  above  at  a  rate 
commensurate  with  the  available  processing  power  while  R; 
is  approximated  by  a  suitably  windowed  a  temporal  aver¬ 
age.  Other  beamforming  methods  such  as  the  maximum 
SINR  method  [23]  may  also  be  used. 


4.1.  Maximum  Likelihood  DOA  Estimation 

The  relationship  between  angle  of  arrival,  beamwidth  of  ar¬ 
riving  signals  and  antenna  spacing  has  been  explored  in  [21] 
and  the  references  therein.  For  an  inter-element  spacing  of 
p  and  narrowband  signal  wavelength  v,  the  fading  experi¬ 
enced  at  adjacent  sensors  is  almost  perfectly  correlated  for 
small  values  of  p/v  (such  as  as  0.5  or  less)  and  angle  spread 
A  (such  as  A  <  10°).  Thus,  inter-element  spacing  p  for 
each  doublet  is  assumed  to  be  such  that  the  two  sensors 
experience  near  identical  fading.  Consider  sensor  m  and 
(m  +  1)  constituting  a  doublet.  Using  (9)  and  (10), 

4?)(P)  =  WLovir^p)  -  wf  (p)Ci^nv[7,(p) 

=  Ak{p)  +  I(k™\p)  (26) 

where  wj t(p)  denotes  the  updated  weight  vector  from  the 
recursions  of  Proposition  3.1;  Iu  \p)  denotes  the  collective 
interference  terms  in  (7)  and  /^(p)  *s  appropriately  de¬ 
fined.  Given  P  observations  from  time  instants  (p  —  P  +  1) 


4.2.  RAKE  Reception 

Consider  now  the  final  stage  of  the  adaptive  array  receiver, 
just  prior  to  the  sheer.  The  Lxl  vector  output  y  *,  (p)  = 
[j/Jto  (p) ,  z/fc  l  (p) ,  ■  ■  ■  ,Vk(L-\){p)]r  of  the  bank  of  beamformers 
is  fed  into  the  RAKE  combiner  to  obtain  symbol  estimates 
for  each  desired  user.  Denote  the  Zcth  users’  RAKE  combin¬ 
ing  coefficients  by  rfc(p)  =  [rfc0(p), rfci(p), . . .  ,r^L-1)(p)]T . 
The  output  of  the  Zcth  users’  combiner  is  then  given  by 

Ak(p)  =  rf  (p)yfc(p)  (31) 

There  are  several  RAKE  combining  algorithms,  the  classical 
ones  being  maximal  ratio,  equal  gain  and  selection  combin¬ 
ing  [12].  Any  one  of  these  methods  or  other  variations  in  the 
literature  [25]  may  be  used.  Such  methods  typically  hinge 
on  the  slowly  fading  assumption  and  make  use  of  special 
pilot  symbols  or  training  sequences  to  update  the  RAKE 
combining  coefficients.  In  a  flat  fading  environment,  the 
RAKE  combining  reduces  to  a  single  1-tap  equalizer  yield- 
ing 

Ak(p)  =  rko{p)yk(p) 


(32) 
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The  equalizer  operates  on  the  received  complex  baseband 
symbol  before  slicing  by  correcting  the  magnitude  and 
phase  of  the  single  beamformer’s  symbol  rate  output. 


5.  SIMULATION  RESULTS 

Consider  first  the  performance  of  the  proposed  receiver  in 
a  flat  Rayleigh  fading  environment.  Fading  coefficients  are 
generated  using  the  standard  Jakes  model  [12].  Figure  3 
depicts  the  ensemble  averaged  signal-to-interference  ratio 
(SIR)  obtained  using  matched  filtering  and  multiuser  detec¬ 
tion  using  OBE,  standard  LMS  and  RLS  algorithms.  The 
latter  three  algorithms  are  used  for  updating  the  GSC’s 
(N  —  1)  x  1  adaptive  weight  vector  component  with  a  single 
signal  preserving  constraint.  Coherent  quadrature  modula¬ 
tion  is  used  with  a  processing  gain  of  N  =  16  with  a  single 
antenna  element  (M  =  1)  and  a  background  SNR  of  20  dB 
due  to  Additive  White  Gaussian  Noise  (AWGN).  Similar 
curves  are  obtained  for  larger  processing  gains.  There  are 
a  total  of  11  active  users  including  the  desired  user,  each 
transmitting  with  equal  power.  Taking  the  processing  gain 
into  account,  this  results  in  an  initial  SIR  value  of  about 
2  dB  representing  a  severe  interference  environment.  The 
LMS  update  gain  is  fixed  at  0.001  while  the  RLS  forgetting 
factor  is  chosen  to  be  0.99.  The  error  specification  for  OBE 
is  set  at  MPq  +  £  where  Pq  is  an  estimate  of  the  desired 
user’s  power  and  £  is  design  parameter  chosen  to  be  unity. 
The  SIR  at  the  pth  symbol  is  calculated  according  to 


SIR(p)  = 


E,  lW"(P)v.U7,r(p)|2 

Er  |w"  (p)v;„t, T(p)\2 


(33) 


where  the  ensemble  average  is  carried  out  over  200  indepen¬ 
dent  trials  in  each  of  which  the  chip  sequences  are  generated 
randomly;  w,.  (p)  denotes  the  detector  weight  vector  in  the 
nth  trial  at  the  pth  symbol;  vsis,r(p)  and  v<nt,r(p)  denote 
the  JVxl  received  signal  and  interference  chip  vectors  in 
the  nth  trial  at  the  pth  symbol  respectively,  w  is  adapted 
using  OBE,  LMS  and  RLS  or  is  fixed  when  the  matched 
filter  is  used. 

An  antenna  array  with  M  =  4  elements  is  introduced 
in  Figure  4  and  the  ensemble  averaged  mean  squared  error 
(MSE)  at  the  receiver  output  is  shown.  It  is  assumed  that 
channel  estimates  are  available  at  the  receiver  to  equalize 
the  pre-slicer  symbol  rate  samples.  Some  degradation  in 
the  achievable  MSE  is  to  be  expected  when  the  estimates 
are  not  perfect.  MSE  with  a  strong  interferer  introduced 
at  the  450th  symbol  is  depicted  in  Figure  5  using  the  RLS 
and  OBE  algorithms.  The  DOA’s  of  all  11  users  are  ran¬ 
domly  distributed  in  the  range  [-60°,  +60°],  each  with  an 
anglespread  of  5°.  An  inter-sensor  spacing  of  p  =  v/8  is 
used.  Perfect  correlation  is  assumed  between  the  fading  ex¬ 
perienced  at  adjacent  sensors.  This  is  known  to  be  a  good 
approximation  for  the  sensor  spacing  and  angle  spreads  un¬ 
der  consideration  [21].  The  DOA-MLE  algorithm,  (27)  is 
used  for  DOA  estimation  and  the  beamformer  weights  are 
obtained  using  the  MVDR  criterion,  (29). 

Figures  6  and  7  show  the  performance  of  the  DOA-MLE 
algorithm  alone.  The  magnitude  and  root-mean-square 
(RMS)  value  of  the  estimation  error  under  different  post¬ 
detection  SNR  ratios  are  shown.  Each  data  point  is  ob¬ 
tained  by  ensemble  averaging  over  1000  independent  trials 


using  a  single  sensor  doublet  with  spacing  p  =  v/8.  In 
Figure  6,  the  angle  spread  A  is  held  fixed  at  0°  while  the 
number  of  symbols  P  used  for  forming  the  estimate  is  var¬ 
ied.  In  Figure  7,  P  —  500  while  A  is  varied.  The  angle 
spread  is  assumed  to  arise  from  multipath  sub-components 
uniformly  distributed  in  the  interval  [— A/2,  A/2]. 

In  the  simulations  results  described  above,  it  is  observed 
that  the  OBE  algorithm  outperforms  the  RLS  algorithm 
in  terms  of  convergence  and  complexity.  Most  of  the  OBE 
updates  are  confined  to  first  few  hundred  symbols  and  the 
update  rate  drops  significantly  after  that.  Overall,  the  per¬ 
centage  of  updates  is  20%  or  less  in  the  simulations  shown 
and  far  lower  when  the  percentage  of  updates  is  computed 
over  a  larger  number  of  symbols.  This  burst  nature  of  up¬ 
dates  has  been  fruitfully  exploited  for  significant  compu¬ 
tational  savings  [8].  Relative  to  OBE  and  RLS,  the  LMS 
algorithm  does  not  perform  adequately  -  a  finding  consis¬ 
tent  with  those  in  the  literature  [16].  In  situations  where 
use  of  the  LMS  algorithm  is  not  subject  to  choice,  the  above 
results  are  suggestive  of  using  an  variable  step  size  LMS  al¬ 
gorithm  which  can  adapt  to  the  time-varying  interference 
conditions.  The  use  an  antenna  array  at  the  base  station 
with  even  2  or  4  elements  is  seen  to  be  very  beneficial  for  DS- 
CDMA  reception,  especially  when  coupled  with  multiuser 
detection. 


6.  CONCLUSIONS 

This  paper  presented  a  new  receiver  structure  for  CCI  sup¬ 
pression  and  fading  compensation  for  CDMA  signaling  over 
frequency-fiat  or  frequency-selective  slowly  fading  channels. 
The  CCI  suppression  and  fading  compensation  are  accom¬ 
plished  by  combining  multiuser  detection,  beamforming  and 
RAKE  reception  in  a  single  integrated  receiver.  Conven¬ 
tional  RAKE  reception  is  used  to  combat  multipath  fading 
while  CCI  suppression  is  carried  out  by  multiuser  detector 
and  beamforming.  ( N  —  1  4-  M)  adaptive  weights  are  used 
for  the  multiuser  detector  and  beamformer,  where  N  is  the 
spread-spectrum  processing  gain  and  M  is  the  number  of 
antenna  array  elements.  The  blind  adaptive  multiuser  de¬ 
tector  is  formulated  using  a  constrained  energy  minimiza¬ 
tion  criterion  and  adaptation  is  carried  out  using  a  novel 
OBE  algorithm.  The  OBE  multiuser  detector  provides  fast 
convergence  and  superior  tracking  relative  to  conventional 
adaptive  algorithms  such  as  LMS  and  RLS.  Also,  a  simple 
and  robust  approximate  Maximum  Likelihood  DOA  esti¬ 
mator  is  presented  for  beamforming. 

APPENDIX 

Spatio-Temporal  Impulse  Response 

The  composite  spatio-temporal  impulse  response  of  the 

channel  and  the  mth  sensor  to  the  fcth  user’s  signals  is  given 

by 

L—  1 

=  (34) 

1=0 

where  a is  the  response  of  the  mth  antenna  element  to 
the  1th  multipath  component  from  the  ftth  user.  Each  mul¬ 
tipath  component  is  received  with  an  angle  spread  of  A  and 
is  assumed  to  be  distinct  from  all  other  paths  of  the  same 
user.  The  angle  spread  arises  due  to  a  large  number  of  rays 
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emanating  from  local  scatterers  in  the  vicinity  of  the  trans¬ 
mitting  source.  Each  RAKE  branch  sees  a  distinct  fading 
signal  with  a  certain  angle  spread.  Thus, 

a(J )  =  Y^PkueKqkli+^)  (35) 


where  the  summation  is  taken  over  all  the  sub-components; 

denotes  the  angular  deviation  of  the  spatial  angle  of 
the  ith  sub-component  of  the  Zth  multipath  component  and 
Pkn  and  qui  denote  the  corresponding  channel  magnitude 
and  phase  response  respectively.  The  spatial  angle  < is 
given  by 

fiku  =  2tt (mp/v)  sin{8ki  +  88ku)  (36) 

where  8ki  denotes  the  A;th  users’  nominal  angle-of-arrival  of 
/th  multipath  component  and  89ku  is  the  angular  deviation 
of  the  ith  sub-component’s  DOA.  For  small  angle  spread  A, 

07) 

where 

<t>  =  2n(mp/v)sm(8ki)  and 

fi&ku  =  2n(mp/v)[cos(8ki)S9kii 

-(1/2)  sin  {8kl)88iu}  (38) 

Note  that  for  point  sources,  8<f> =  0.  Using  (35)-(38), 

—  pkl  ie?qk,i  e^^ku  (39) 


(^JiPkiieiqk,i e?S't’k,i  )  is  usually  modeled  as  a  complex 
Gaussian  random  variable  since  it  is  the  summation  of  a 
large  number  of  i.i.d.  random  variables  constituting  the 
channel  attenuation  for  each  multipath  component. 

DOA-MLE  Derivation 

Rewriting  (26)  in  vector  form  using  P  samples  for  sensor  m 
and  (m  +  1), 


e'75  =  ^Pk^Ak+lif  (40) 

efcT+1)  =  VK^kT+1)M  +  iiT+1)  (4i) 

where  ej/,n\  a^”1* ,  A*,  and  denote  vectors  of  length  P 
of  the  respective  temporal  samples.  Using  (39), 


a 


(m+l) 

kl 


J<t> 


k‘  a 


(m)  _j_  £(m+i) 


kl 


kl 


(42) 


where  ej^  denotes  the  residual  difference  between  the  chan¬ 
nel  attenuation  at  the  with  sensor  due  to  angle  spreading. 
To  obtain  a  series  expansion  of  (39)  may  be  carried 
out  for  small  8cj> according  to 


*(m) 
?kl  i 


1  +j'C 


(m) 


Thus,  yielding 

4?*  =  e5**'  '  '52U$<l>ki)i)Pkiiejqk‘ 


(43) 


(44) 


Under  isotropic  scattering,  can  be  regarded  as  complex 
valued  zero  mean  and  Gaussian  distributed.  Denoting  (p^ 
by  $  for  notational  simplicity  and  the  conditional  probabil¬ 
ity  density  function  of  e^"'’  by  fe,  4 >ml  is  given  by 

’  $mi  =  arg  max/e(e^7+1)>el7V^’) 

=  argmax/e(e[7+1)/ei7)>^)  (45) 

since  is  independent  of  $.  Now,  using  (41)  and  (42), 
e<7+1)  =  e<7V* -i£*V*  +i(J+1)  +el?+1)VPkAk  (46) 

Conventional  assumptions  are  now  made  on  the  post¬ 
detection  interference  vectors  to  enable  use  of  a  ML  ap¬ 
proach.  Namely,  they  are  assumed  to  be  instances  of  a  sta¬ 
tionary,  ergodic  zero-mean  complex  valued  Gaussian  pro¬ 
cess.  Thus,  the  mean  of  e[,7+1'  is  (e^e3*)  and  fe  can 
be  factored  into  a  product  of  partial  densities.  Thus,  Ijui 
can  be  obtained  by  maximizing  the  log  likelihood  function 
according  to 


&ml  =  arg  max  (e<7+1)  -  (el7+1)  - 

(47) 

Differentiating  the  right-hand  side  of  (47)  with  respect  to 
and  setting  to  zero  yields 


fel7)lHei7+1)e-^  =  [e[7+1)lffe[7)e^ 


(48) 


Noting  that  the  right-hand  side  of  (48)  is  merely  the  com¬ 
plex  conjugate  of  the  left-hand  side,  and  setting  the  imagi¬ 
nary  part  to  zero  yields  the  desired  result  in  (27). 
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Figure  2.  Schematic  depicting  operation  of  OBE 
recursions  in  two  dimensions. 


Figure  4.  MSE  using  MF,  OBE  and  RLS  algo¬ 
rithms.  Processing  gain  N  —  16,  M  =  4. 


Figure  3.  SIR  using  MF,  OBE  and  RLS  algorithms. 
Processing  gain  N  =  16,  M  —  1. 


Figure  5.  MSE  using  MF,  OBE  and  RLS  algorithms 
with  strong  interferer.  Processing  gain  N  —  16,  M  = 
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Figure  6.  Magnitude  and  RMS  value  of  DOA  es¬ 
timation  error  using  DOA-MLE  algorithm  (in  de¬ 
grees):  Number  of  symbols  P  versus  SNR  for  angle 
spread  A  =  0. 
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Figure  8.  Receiver  structure  for  L-path  frequency 
selective  fading  channel  for  fcth  user. 
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Figure  7.  Magnitude  and  RMS  value  of  DOA  es¬ 
timation  error  using  DOA-MLE  algorithm  (in  de¬ 
grees):  Angle  spread  A  versus  SNR  for  number  of 
symbols  P  =  500. 
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1.  SUMMARY 

The  coming  of  years  of  high  solar  flux,  starting 
in  1999,  will  bring  propagation  problems  to 
some  users  of  GPS;  we  are  concerned  with 
ionospheric  irregularities  which  produce  phase 
and  amplitude  scintillation.  The  knowledge  of 
when  deteriorated  signals  will  occur  and  their 
characteristics  can  help  minimize  their  impact 
or  at  least  allow  users  to  understand  that  the 
fading  phenomena  are  of  natural  origin  rather 
than  equipment  related.  At  equatorial  anomaly 
latitudes,  fades  of  20  dB  at  GPS  frequencies 
have  been  observed  in  years  of  high  solar  flux; 
according  to  receiver  experts  these  are  enough 
to  disturb  any  equipment.  The  sophistication  of 
the  equipment  in  moving  through  these  signal 
problems  is  a  function  of  hardware  and 
complexity  of  software.  Knowledge  of  diversity 
can  help  to  minimize  the  effect  of  temporary 
loss  of  signal. 

The  equatorial  region  irregularity  structures  are 
dominated  by  patches  of  irregularities  which  are 
frequently  of  the  order  of  1 00-200  km  in  their 
East  West  extent.  At  times  these  extensions  may 
be  exceeded.  In  addition  their  effective 
magnetic  North  South  dimensions  may  be 
effective  to  a  few  thousand  kilometers  along  the 
earth's  magnetic  field.  If  the  altitude  that  the 
plume  of  irregularities  reached  at  the  magnetic 
equator  is  high,  the  effects  reach  the  anomaly 
region  with  its  high  electron  density  and  show 
strong  scintillation  intensities.  New  data  on  the 
variety  of  altitudes  of  irregularities  at  the 
equator  puts  limits  on  forecasting  of  some 
structures  but  hope  for  forecasting  some  major 
plumes. 


In  the  polar  region  severe  problems  are  not 
expected  to  surface  until  1999  since  polar 
irregularities  have  been  found  to  be  a  function  of 
solar  flux.  In  this  area  diversity  of  propagation 
paths  can  play  an  important  role. 

The  irregularity  region  at  auroral  latitudes  is 
primarily  affected  by  individual  magnetic  storms 
which  can  occur  during  any  phase  of  the  sunspot 
cycle.  Irregularity  intensity  is  a  function  of  various 
parameters  of  the  magnetic  storm  relative  to  that 
station.  Illustrations  of  GPS  phase  fluctuations  will 
be  shown  as  individual  storms  develop.  New  data 
clearly  show  the  forecasting  possibilities  when 
using  timing  of  the  auroral  oval  occurrence  at  a  site 
as  a  means  of  prediction. 

L  Band  is  to  be  used  in  the  downlink  of  new 
systems  proposed  and  being  initiated  for  personal 
communications.  Low  link  margins  are  forecast  for 
several  systems  to  be  in  the  range  6  to  10  dB. 

These  levels  indicate  that  fading  due  to  the 
ionospheric  irregularities  such  as  discussed  in  these 
GPS  studies  will  cause  problems. 

2.  PREFACE 

In  previous  reports  to  AGARD  on  propagation 
problems  with  the  reception  of  the  1 .2  and  1 .6  GHz 
signals  from  the  Global  Positioning  System's  24 
satellites,  we  outlined  several  areas  where 
difficulties  could  be  encountered.  The 
observational  problems  outlined  were  in  the  area  of 
ionospheric  irregularities  and  variations  of  total 
electron  content. 

How  the  receiver  deals  with  propagation  problems 
is  a  function  of  the  hardware-software  combination 
of  particular  designs.  When  only  simple  single 
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frequency  receivers  are  used,  there  could  be 
problems  ranging  from  a  long  period  being 
necessary  to  acquire  the  satellite  signal  from  a 
cold  start  to  a  loss  of  lock  due  to  deep  amplitude 
or  phase  scintillations.  Erratic  readings  could 
result  during  large  swings  of  fluctuations.  The 
complexity  of  the  receiver  can  be  such  that 
flywheel  aspects  can  essentially  reduce  or 
eliminate  certain  groups  of  fading  problems; 
equipment  parameters  have  to  be  folded  into  the 
problem  (Clynch  and  Aarons,  1996).  However 
in  differential  GPS,  phase  fluctuations  have  to 
be  evaluated. 

The  total  electron  content  variations  of 
importance  in  accurate  measurements  include 
horizontal  gradients  both  east-west  and  north- 
south  as  well  as  travelling  ionospheric 
disturbances  of  various  types. 

3.  SCINTILLATION  OCCURRENCE 

While  the  general  patterns  of  scintillation  have 
been  given  in  the  literature  and  at  AGARD 
Symposia,  recent  studies  have  advanced  the 
ability  to  forecast  high  latitude  occurrence.  In 
the  field  of  forecasting  equatorial  scintillation 
there  can  be  methods  of  forecasting  some  major 
disturbances  as  well  as  localized  turbulence.  In 
any  event  the  understanding  of  the  problems 
and  the  propagation  will  allow  users  to  correctly 
identify  the  behavior  of  the  system  as  due  to 
trans-ionospheric  propagation. 

4.  DATASET 

The  data  set  for  these  studies  is  obtained 
through  the  International  GPS  Service  for 
Geodynamics  (IGS).  Using  these  data  we  can 
know  the  patterns  of  auroral  fluctuations  and 
equatorial  effects  in  a  more  precise  manner  than 
earlier  studies.  The  many  stations  reporting 
daily  data  (over  a  hundred)  has  allowed  for  the 
first  time  a  comparison  of  phase  fluctuation 
over  large  areas.  Thus  the  development  of 
irregularities  whether  proceeding  equatorward 
in  the  high  latitude  region  or  moving  westward 
with  sunset  in  the  equatorial  region  is  now 
better  understood. 


Phase  fluctuations  reported  in  the  present  paper 
were  obtained  by  examining  30  second  values  of 
total  electron  content  (TEC).  Each  30-s  data  point 
is  obtained  from  the  phase  differences  between  the 
1 .2  GHz  and  the  1 .6  GHz  signals  of  each  GPS 
satellite.  The  rate  of  change  of  these  values  is  the 
source  of  the  phase  fluctuations  reported. 

5.  HIGH  LATITUDE  REGION 

It  has  been  known  that  during  magnetic  storms, 
irregularities  develop  in  the  auroral  oval  of  an 
intensity  greater  than  during  magnetically  quiet 
periods.  During  magnetic  storms  the  irregularity 
oval  expands  both  equatorward  and  poleward  -  and 
the  fluctuations  intensify.  A  large  amount  of  data 
from  1993  to  1997  has  been  studied.  The  data  are 
processed  in  the  manner  illustrated  in  Figures  la 
and  lb.  For  a  high  latitude  station  (Tromso)  we 
show  an  example  of  the  data  input  and  a  first  look 
at  the  observations.  Figure  1  a  is  a  map  of  the 
region  in  Corrected  Geomagnetic  Latitude  and 
Longitude  traversed  by  the  satellite  propagation 
path  at  400  km.  The  total  electron  content  is  plotted 
in  the  middle  panel.  The  rate  of  change  of  total 
electron  content  taken  from  the  phase  differences 
between  1 .2  GHz  and  1 .6  GHz  is  shown  on  the 
right.  This  is  not  a  very  intense  magnetic  storm. 

For  the  station  and  the  equipment  taking  the  data 
the  fluctuations  were  not  enough  to  cause  loss  of 
lock.  In  Figure  lb  we  show  two  satellite 
propagation  paths  and  the  resulting  phase 
fluctuations  as  noted  from  Santiago,  Chile,  a 
station  in  the  equatorial  anomaly  region. 

When  a  storm  develops,  irregularities  are  produced 
in  the  auroral  region.  With  increasing  magnetic 
indices  the  auroral  region  expands  equatorward  and 
poleward  and  the  irregularities  are  both  more 
frequent  and  of  higher  intensity.  This  can  be  seen 
in  the  illustration  of  Figure  2  where  both  days  of 
low  and  high  magnetic  indices  are  shown.  Tromso 
and  Kiruna  show  high  intensities  for  the  magnetic 
storm  of  September  27-28.  Ny  Alesund 
(Spitzbergen)  above  the  auroral  oval  shows  some 
increase  but  does  not  intensify  the  way  phase 
scintillations  do  over  Tromso  and  Kiruna;  Kiruna  is 
approximately  1.5°  lower  in  corrected  geomagnetic 
latitude  than  Tromso. 
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6.  EQUATORIAL  REGION 

The  map  of  Figure  3  contains  the  geographic 
positions  of  many  of  the  stations  used  in  this 
study.  At  the  anomaly  latitude  of  Santiago  we 
have  extremely  high  total  electron  content  and 
F2  electron  density  during  certain  times  of  the 
day  when  compared  to  observations  at  the 
magnetic  equator.  In  a  campaign  in  October 
1 996,  one  magnetic  storm  produced,  at  the 
South  American  longitudes,  irregularities  across 
the  continent.  GPS  stations  in  Fortaleza,  Brazil, 
La  Plata,  Argentina,  Kourou,  French  Guiana, 
Bogota,  Columbia,  and  Arequipa,  Peru  all 
showed  several  hours  of  phase  scintillations  in 
the  post  sunset  to  midnight  time  period,  similar 
to  that  shown  in  Figure  lb.  This  was  not  a 
frequent  occurrence  in  this  solar  minimum  year. 
During  the  month  of  October  1996,  only  two 
nights  showed  scintillation  activity  at  Santiago 
in  this  sunspot  minimum  year. 

Figure  lb  showed  data  from  two  satellites 
observed  from  Santiago  when  a  high  altitude 
plume  developed  at  the  magnetic  equator.  Each 
of  the  two  satellites  showed  high  levels  of  phase 
scintillations  but  there  were  periods  of  low  level 
fluctuations  indicating  diversity  path 
possibilities.  The  data  were  taken  in  the 
magnetic  storm  period  of  October  22-23,  1996. 
For  the  anomaly  region  there  was  a  bifurcated 
structure  to  the  irregularity  region  as  viewed  by 
Satellite  15  (02-04  UT).  This  allows  periods  of 
time  when  no  scintillations  would  be  observed 
on  an  individual  path.  In  Figure  lb  for  example 
at  0230  UT  on  the  path  to  Satellite  1 5,  no 
scintillation  activity  is  observed.  However 
phase  fluctuations  are  noted  on  the  path  to 
Satellite  14  at  that  time.  It  can  be  seen  that  the 
level  of  fluctuations  did  not  produce  loss  of  lock 
on  either  path  for  the  receiver  used  in  Santiago 
measurements. 

Total  electron  content  in  the  post  sunset  period 
in  the  anomaly  region  increases  considerably 
during  years  of  high  solar  flux.  The  ratio  of  total 
electron  content  (solar  maximum  compared  to 
solar  minimum)  using  model  calculations  has 
been  made  for  the  sunset  time  period  by  M.  Fox 
(personal  communication).  For  TEC  the  model 
indicates  well  over  a  factor  of  5  during  the 


sunset  time  period  when  maximum  solar  flux 
periods  are  compared  to  minimum  solar  flux 
periods.  At  L  Band,  Basu,  et  al.,  1988  has  found 
fades  reaching  peak  to  peak  values  of  over  20  dB  in 
the  anomaly  region  in  years  of  high  solar  flux; 
fading  of  this  magnitude  can  last  for  several  hours 
as  has  been  shown  even  in  4  GHz  results  (Fang  and 
Liu,  1984).  In  years  of  low  solar  flux  only  short 
lived  periods  of  10  dB  peak  to  peak  values  have 
been  reported.  The  question  remains  whether 
simple  receivers  will  lose  lock  when  solar  flux 
increases  in  the  coming  years. 

It  is  of  interest  that  during  years  of  low  solar  flux, 
several  magnetic  storms  have  produced 
disturbances  at  both  equatorial  and  auroral 
latitudes.  This  may  or  may  not  be  the  case  during 
high  solar  flux  years.  An  example  of  one  storm's 
effects  at  equatorial  and  auroral  latitudes  is  shown 
in  Figure  4  where  both  Yellowknife  at  high 
latitudes  shows  intense  scintillations  as  does 
Santiago  in  the  anomaly  region.  Auroral  latitudes 
lower  than  the  position  of  Yellowknife  show 
considerably  lower  occurrence  of  intense  phase 
scintillations.  This  storm  with  its  high  intensity 
lasting  over  a  period  of  hours  at  anomaly  latitudes 
is  not  the  only  storm  of  its  kind  found  in  recent 
studies.  A  series  of  magnetic  storms  is  shown  with 
similar  effects  in  Aarons  et  al.,  1996.  In  particular 
the  storm  of  October  22-23,  1996  was  similar  to  the 
storm  of  October  3, 1994;  radar  backscatter  from 
the  October  3,  1994  storm  is  shown  in  Figure  5 
from  Aarons  et  al.,  1996.  In  this  case  irregularities 
developed  over  many  hours. 

It  should  be  noted  that  systems  being  proposed  for 
personal  communications  use  L  Band  (Evans, 

1997).  Fading  problems  will  certainly  occur  and  for 
the  anomaly  region  (which  includes  cities  such  as 
Hong  Kong,  Santiago,  Bogota)  there  will  be  fading 
problems  in  the  hours  between  sunset  and 
midnight.  Margins  for  these  systems  only  range 
from  6  to  16  dB. 

7.  CONCLUSIONS 

The  spreading  of  irregularity  development  during 
magnetic  storms  at  high  latitudes  is  ubiquitous  in 
the  magnetic  midnight  sector.  Areas  equatorward 
and  poleward  of  the  auroral  oval  are  affected. 
However  in  the  equatorial  anomaly  regions  the 
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high  altitudes  of  the  plumes  show  splitting  or 
bifurcation  and  irregularities  can  move  past  a 
disturbed  propagation  path;  there  are  paths 
which  are  relatively  free  of  scintillations. 

The  ability  of  a  particular  GPS  receiver  to  glide 
through  phase  and  amplitude  scintillations  is  a 
function  of  the  hardware  (signal  to  noise  levels) 
and  the  software  dealing  with  problems  of  loss 
of  signal  i.e.  the  flywheel  effect.  In  the  coming 
solar  maximum,  high  latitude  paths  which  show 
intense  phase  and  amplitude  scintillation  will 
encompass  the  polar  region;  data  are  not 
available  to  determine  if  the  irregularities  which 
develop  in  the  auroral  region  will  reach  greater 
intensity  during  years  of  solar  maximum  than 
during  years  of  solar  minimum.  The  knowledge 
of  the  effects  of  the  ionosphere  on  transmissions 
can  be  of  importance  to  users  in  the  field. 
Knowledge  of  the  phenomena  will  help  to 
minimize  effects  and  to  understand  that 
observational  problems  may  be  due  to 
ionospheric  effects  rather  than  to  equipment 
failures. 
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TROMSO 

UT  Date:  September  27,  1995 


Figure  la.  Trornso  data  for  one  satellite  includes  a  map  in  Corrected  Geomagnetic 
Latitudes  with  the  Trornso  site  shown  by  an  X.  Total  electron  content  and  rate  of  change 
of  total  electron  content  are  shown  in  the  next  two  panels. 


SANTIAGO 

UTDate:  October  23,  1996 


Figure  lb.  Data  similar  to  that  of  Figure  la  are  shown  for  Santiago,  Chile.  There  is  an 
added  fourth  panel  of  relative  intensity  of  phase  fluctuations. 
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PHASE  FLUCTUATIONS 

SEPT.  23  -  OCT.  2,  1995 
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Figure  2.  Scintillation  activity  of  various  levels  are  shown  for  the  quiet  period  before  the 
magnetic  storm  of  September  27-28,  1995  and  during  the  magnetic  storm.  Various  levels 
are  indicated  where  the  rate  of  change  of  Total  Electron  Content  Units  exceeds  plus  and 
minus  .5  TECU/minute.  A  Total  Electron  Content  Unit  is  1  x  1 0 1 6  electrons  per  square 
meter. 
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High  Latitude  GPS  Stations 
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Figure  3.  A  map  for  several  of  the  stations  used  for  this  study. 
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The  Magnetic  Storm  Period  of  November  2-6, 1993 


NOVEMBER  2 -6, 1993 
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SANTIAGO 
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NOVEMBER  2 -6,  1993 
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Figure  4.  Magnetic  storm  effects  increased  occurrence  and  intensity  of  scintillation 
activity  at  high  latitudes  (Yellowknife)  but  less  at  St.  John's  at  a  lower  latitude.  At  the 
equatorial  anomaly  latitudes  of  Santiago  and  Kourou  high  scintillation  activity  is  shown 
for  the  magnetic  storm  day  of  November  3-4,  1993.  For  the  quiet  day  of  November  2-3, 
Kourou  at  a  latitude  closer  to  the  magnetic  equator  than  Santiago  shows  phase 
fluctuations  but  Santiago  at  18°  dip  latitude  does  not.  It  should  be  noted  that  equatorial 
irregularities  occur  during  both  magnetically  quiet  and  magnetically  disturbed  periods. 
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Figure  5.  Backscatter  returns  on  the  magnetic  storm  day  of  October  3,  1994  as  observed 
at  Jicamarca,  Peru  (Aarons  et  al.,  1996).  A  plume  similar  to  this  in  duration  and  altitude 
also  occurred  on  October  22-23,  1996;  data  shown  in  Figure  lb  are  from  that  storm 
period. 
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PAPER  No:  24 

DISCUSSOR’S  NAME:  R.  Sabatini 

COMMENT/QUESTION: 

1 .  You  mentioned  that  TEL  fluctuations  have  a  very  important  effect  on  GPS  performance. 
Probably  this  is  not  true  for  P-code  double  frequency  receivers.  Is  that  correct? 

2.  Are  there  any  substantial  differences  between  the  performance  of  DGPS  using  “carrier- 
phase”  and  DGPS  using  “pseudo  range"  with  regard  to  atmospheric  propagation  effects? 


AUTHOR/PRESENTER’S  REPLY: 

1 .  Yes,  military  P-code  receivers  are  not  affected  by  TEL  fluctuations. 

2.  We  did  not  investigate  this  particular  aspect. 
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METEOROLOGICAL  DATA  FUSION  TO  ASSESS  EM/EO  PROPAGATION  CONDITIONS  IN 

COASTAL  ENVIRONMENTS 

J.  Rosenthal 
R.  Helvey 
P.  Greiman 
M.  McGovern 

Geophysics  Branch  (521400E) 

Naval  Air  Warfare  Center  Weapons  Division 
Point  Mugu,  California  93042-5001  USA 


SUMMARY 

EM/EO  propagation  conditions  can  be  estimated 
along  specific  routes  or  regions  of  interest  by  fusing 
on  one  display,  the  visible  satellite  imagery  showing 
cloud  cover,  the  operational  geometry,  and  derived 
quantities  such  as  duct  height  estimated  from  the  IR- 
duct  technique.  In  addition,  by  compositing 
conditions  over  a  set  of  similar  synoptic  events,  and 
when  also  fused  with  averaged  synoptic  field  gridded 
data,  an  average  depiction  of  duct  height  and 
associated  variability  can  be  obtained.  To  the  extent 
the  cases  selected  are  typical  of  that  type  of  event,  the 
averaged  picture  can  be  used  to  forecast  or  estimate 
the  distribution  of  EM/EO  parameters  in  the  near 
future  if  confidence  exists  in  the  forecast  synoptic 
situation.  Additional  compositing  is  underway  to 
broaden  the  range  of  synoptic  weather  types  and 
regions  examined. 

BACKGROUND 

Efforts  have  continued  at  the  NAWCWTNS 
Geophysics  Branch  at  Point  Mugu  to  improve  the 
automation  and  accuracy  of  satellite-based  efforts  to 
diagnose  the  character,  height,  strength  and  horizontal 
variability  of  EM/EO  propagation  conditions. 
Rosenthal  et  al  (Ref.  1,  2)  laid  the  groundwork  for 
some  of  these  efforts  using  descriptive  and  semi¬ 
objective  techniques  of  cloud  pattern  recognition  of 
stratus/stratocumulus  clouds  to  infer  information  on 
the  distribution  of  refractive  properties.  Lyons  (Ref. 
3,  4)  and  Helvey  (Ref.  5,  6,  7)  developed  the  more 
automated  "Infrared  (IR)  Duct"  technique  to  convert 
satellite-derived  cloud-top  temperatures  to  an 
estimate  of  duct  height,  and  Greiman  and  Rosenthal 
et  al  (Ref.  8)  have  attempted  to  use  the  approach  to 
measure  the  amount  of  horizontal  variability  in  the 
marine  boundary  layer.  Utility  of  these  procedures 
for  discerning  some  of  the  atmospheric  features 
important  to  electro-optical  (E-O)  systems  has  also 
been  evident. 

Both  the  subjective  and  objective  satellite  techniques, 


together  with  synoptic  and  climatological  conditions 
are  being  integrated  or  fused  together  in  an  expert 
system  (EXPERDUCT)  to  allow  different  levels  of 
user  expertise  to  synthesize  various  factors  in  making 
refractive  assessments  (Ref.  7) 

This  paper  summarizes  some  of  the  more  recent 
efforts  to  determine  variability  by  employing  the  IR- 
duct  and  statistical  techniques. 

HORIZONTAL  VARIABILITY  ASSESSMENTS 

Several  papers  (Rosenthal,  Helvey,  Greiman  (Ref.  7, 
8))  have  documented  use  of  the  IR-Duct  approach  to 
measure  the  horizontal  variability  of  duct  heights 
across  specific  synoptic/mesoscale  features.  By  using 
this  approach  for  the  same  region  and  time  on 
successive  days,  a  measure  of  the  temporal  changes 
associated  with  these  case  studies  was  observed  as 
well.  Though  of  significant  value  as  a  diagnostic  tool 
for  EM/EO  assessment,  (see  fig.  1)  the  unique 
features  of  each  case  study  make  the  approach  of 
using  IR-duct  heights  for  predictive  purposes 
dependent  on  the  extent  to  which  the  case  study 
approximates  the  typical  dimensions  and  intensities 
for  that  type  of  feature. 

In  order  to  generalize  the  distribution  of  duct  heights 
that  can  be  expected  for  different  types  of  synoptic 
regimes,  an  averaging  technique  has  been  employed 
which  permits  the  characteristics  of  several  similar 
synoptic  cases  to  be  lumped  or  composited  together. 

COMPOSITING  TECHNIQUE 

The  methodology  for  this  involves  initial  selection 
of  case  studies  for  a  group  of  similar  meteorological 
patterns  observed  over  the  western  U.  S.  and  Eastern 
Pacific  Ocean,  e.g.  Santa  Ana  (Great  Basin  High 
Pressure/offshore  wind  flow);  cut-off  low  pressure 
systems;  stable  ridge,  or  other  common  synoptic 
situation.  U.  S.  Department  of  Commerce's  "The 
"Daily  Weather  Map"  weekly  series  (Ref.  9)  was  used 
to  select  common  episodes.  Using  a  commodore 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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Amiga  3000,  gridded  U.  S.  National  Center  for 
Environmental  Prediction  (NCEP)  Medium  Range 
Forecast  (MRJF)  1000  MB,  700  MB  and  500  MB 
height  data  and  concurrent  GOES-9  satellite  imagery 
was  down  loaded  from 

1  GB  'JAZ'  diskettes  onto  floppy  disks  for  display  on 
an  IBM  PS/2  Model  70  computer  as  part  of  the 
Meteorological  Interactive  Data  Display  System 
(MIDDS).~ 

Duct  heights  were  calculated  over  areas  covered  by 
low  stratus/stratocumulus  clouds  using  the 
NAWCWPNS  IR-duct  technique  along  selected 
radials  (or  arcs)  spanning  the  domain  of  analysis. 

The  domains  for  both  satellite  and  field  data  were 
synchronized.  Generally,  four  to  six  'radials'  were 
sufficient  to  cover  the  domain,  depending  on  cloud 
cover,  with  three  to  six  points  along  each  radial 
(again  dependent  on  cloud  cover  at  designated  pixel 
locations). 

After  all  duct  heights  were  calculated,  they  were 
averaged  for  each  point  on  the  radial.  This  generated 
average  duct  heights  for  each  latitude-longitude  point 
on  each  radial  for  the  selected  cases. 

Next,  MRF  gridded  field  data  were  also  averaged  and 
displayed  on  the  MIDDS  system,  together  with  the 
calculated  duct  heights. 

RESULTS 

A  composite  of  gridded  field  data  at  two  levels  and 
satellite-derived  IR-Duct  heights  is  shown  in  figure  2 
for  what  has  been  termed  a  "San  Diego  Ridge”  -  a 
situation  whereby  strong  ridging  aloft  prevails  over 
the  U.  S.  Southwest  with  a  high  centered  just  to  the 
west  of  San  Diego.  The  averaged  height  contours  at 
500  mb  are  shown  in  figure  2  at  20  meter  intervals. 
Also  shown  are  the  averaged  1000  mb  heights  at  10 
meter  intervals.  The  bold  numbers  are  averaged  duct 
heights  in  feet  (taken  as  the  optimum  coupling  height 
(OCH))  derived  for  the  GOES -9  IR  data.  The  duct 
heights  have  been  manually  analyzed,  -  shown  as 
dashed  lines  or  contours. 

The  period  of  record  used  in  this  composite  covers  3 
episode  periods,  May  10-12,  1996;  June  1  1996;  and 
August  28-30,  1996. 

Inspection  of  this  composite  reveals  a  subsidence- 
induced  minimum  in  duct  height  off  the  Central  and 
Southern  California  Coast  with  minimum  of  about 
600  feet  west  of  the  Monterey  Bay  area. 


Duct  heights  for  this  type  of  synoptic  occurrence 
increased  slightly  to  the  south  and  southeast,  but 
increase  sharply  to  the  north  and  west  where  average 
500  mb  heights  are  much  lower,  and  where 
subsidence  is  presumably  much  reduced.  The  full 
range  of  duct  height  increase  from  the  minimum  to  a 
region  about  300  to  400  miles  away  was  about  2,600 
feet. 

Figure  3  shows  a  composite  for  a  different  synoptic 
situation;  -  one  dominated  by  a  cut-off  low  over  the 
U.  S.  Southwest  with  cyclonic  curvature,  instability 
and  a  minimum  of  subsidence  over  the  Southern 
California  region.  This  figure  was  obtained  by 
averaging  similar  weather  patterns  from  March  12, 
1996;  March  14-15  1996;  May  24-25  1996;  Oct  26, 
28-29  &  31,  1996;  and  Nov  1-3,  1996. 

In  this  kind  of  synoptic  situation,  highest  duct  heights 
(approximately  3,300  feet)  occur  in  a  zone  from  the 
coast  south  of  Monterey  Bay,  extending  seaward  to 
the  southwest  for  approximately  300  miles.  To  the 
northwest  from  this  maximum,  duct  heights  decrease 
by  approximately  1400  feet  in  about  300  miles  to  a 
duct  height  minimum  located  off  the  Northern 
California  coast.  This  marks  the  region  where 
offshore  flow  predominates  north  of  the  low  aloft, 
with  subsidence  of  air  from  inland  areas  to  the  coast. 

It  is  interesting  to  note  that  whereas  the  synoptic 
conditions  are  dramatically  different  in  the  two 
examples  shown,  both  reveal  a  sharp  slope  in  duct 
height  along  the  coast  from  southeast  to  northwest, 
but  of  opposite  sign. 

Additional  synoptic  regimes  are  also  being  examined 
to  characterize  the  variability  of  EM/EO  parameters 
associated  with  each  type.  The  additional 
compositing  should  broaden  the  range  of  synoptic 
weather  types  and  geographical  regions  examined. 

FUSION  OF  SATELLITE  IMAGERY  AND 
OPERATIONAL  GEOMETRY 

Another  useful  tool  that  is  permitted  by  the  Navy's 
current  meteorological  and  oceanographic  (METOC) 
architecture  is  the  ability  to  combine  satellite  imagery 
with  defined  areas  or  lines  of  interest  in  such  a  way 
that  the  user's  attention  can  be  focused  on  cloud  and 
other  conditions  in  his  or  her  immediate  purview. 
Figure  4  shows  an  example  demonstrating  this  simple 
technique.  When  combined  with  other  information 
such  as  estimates  of  duct  height,  a  real-time 
assessment  can  be  made  of  expected  propagation 
conditions  along  specific  routes  that  apply  to  both 


EM  (duct  height)  and  EO  (clouds  and  fog). 
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Figure  2.  Composite  of  averaged  IR-Duct  heights  in  feet  (bold  numbers)  along  selected  radials  superimposed  on 
averaged  height  fields  at  500  mb  and  1000  mb  for  a  "San  Diego  Ridge",  a  feature  that  typically  results 
in  subsidence  and  low  ducts.  Analysis  of  estimated  duct  heights  shown  as  broken  lines. 


Figure  3.  Composite  of  averaged  conditions  (same  as  above)  except  for  synoptic  events  marked  by  cutoff  low 
pressure  system  aloft. 


Figure  4.  Satellite  image  superimposed  by  estimates  of  duct  height  and  by  routes  of  operational  interest  fused  into 
one  display. 
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Resume 

Nous  proposons,  dans  cet  article,  une  methode 
permettant  de  combiner  des  informations  symboliques 
avec  des  informations  numeriques.  Pour  ce  faire,  nous 
proposons  un  processus  d'estimation  supervisee  par  un 
niveau  d'analyse  du  contexte.  L'application  visee  est  la 
fusion  de  donnees  et  les  algorithmes  qui  sont  implantes 
dans  les  systemes  multicapteurs.  Ces  algorithmes 
doivent  etre  congus  pour  que  le  systeme  fonctionne 
d'une  fagon  nominale  dans  toutes  les  conditions.  Pour 
cela  le  systeme  devra  s'adapter,  de  sorte  qu'a  tout  instant 
l'estimation  tienne  compte  du  contexte  considere.  Le 
resultat  etant  de  privilegier,  dans  certains  contextes,  les 
mesures  issues  des  capteurs  en  etat  nominal  de 
fonctionnement  et  de  reduire  l'importance  de  celles  qui 
sont  aberrantes  au  vu  de  certains  criteres  etablis  au 
prealable  par  un  expert. 


1.  Introduction 

Les  developpements  recents  des  systemes  de  perception 
convergent  actuellement  vers  l'utilisation  conjointe  de 
capteurs  multiples  [1],  En  effet,  les  benefices  attendus 
sont  prometteurs  :  une  capacite  plus  importante 
d'analyse  des  situations  complexes,  une  robustesse 
accrue  a  l'environnement.  Les  domaines  concernes 
touchent  aussi  bien  le  milieu  industriel  pour  les  taches 
d’assemblage,  la  robotique  mobile,  que  le  milieu 
militaire  dans  le  domaine  du  commandement  et  du 
controle  de  champs  de  bataille,  de  la  poursuite  de  cibles, 
ou  de  la  navigation  d'engins  aeriens.  L'integration  et  la 
fusion  d'informations  multiples  sont  devenues  des  lors 
une  voie  d'investigation  et  de  recherches  tres  actives. 

Si  les  techniques  classiques  d'estimation  et  de 
classification  basees  sur  la  theorie  des  probabilites  ont 
vu  leur  champ  d’application  s'elargir  a  la  fusion  de 
donnees,  l'emergence  de  problemes  nouveaux  et  propres 
aux  systemes  multicapteurs  a  ete  a  l'origine  de 
recherches  de  nouvelles  modelisations  pour  le  traitement 
de  l'information.  Parmi  celles-ci  la  logique  floue  [6]  et  la 
theorie  de  Dempster-Shafer  [9]  sont  apparues  comme  des 
alternatives  aux  probabilites  sans  pour  autant  les 
remplacer.  Elies  permettent  de  manipuler  et  de  traiter 
des  informations  souvent  heterogenes  et  d'origines 
incertaines.  De  fagon  generique,  si  la  theorie  des 
probabilites  est  notamment  utilisee  pour  la  modelisation 


de  phenomenes  aleatoires,  la  logique  floue  elle,  a  un 
champ  d'applications  tourne  vers  la  representation  de  la 
connaissance  humaine.  Une  dualite  apparait  done  entre 
ces  deux  theories  qui  peut  etre  mise  a  profit,  notamment 
dans  un  systeme  multicapteur.  En  effet,  les  informations 
manipulees  obeissent  a  cette  meme  dualite,  les  unes, 
aleatoires,  sont  les  mesures  issues  de  chaque  capteur,  les 
autres  relevent  d'une  connaissance  plus  symbolique,  et  se 
revblent  utiles,  de  la  conception  du  systeme  jusqu'a  son 
utilisation. 

Dans  le  domaine  de  la  poursuite  de  cibles  [2],  [3],  [6], 
[7],  [10],  les  algorithmes  proposes  pour  la  fusion  de 
donnees  sont  bases  sur  une  approche  exclusivement 
probabiliste.  Les  filtres  de  Kalman  developpes  et  leurs 
extensions  IMM  (Interacting  Multiple  Models),  PDAF 
(Probability  Data  Association  Filter),  JPDAF(Joint 
Probability  Data  Association  Filter),  ...  [2],  supposent 
que  les  capteurs  qui  composent  le  systeme  ont  des 
caracteristiques  connues.  En  outre,  le  contexte  n'est 
jamais  pris  en  compte,  ce  qui  suppose,  implicitement, 
que  celui-ci  est  favorable  a  l'utilisation  simultanee  de 
l’ensemble  des  capteurs.  Cette  hypothese  est,  a 
l'evidence,  souvent  tres  loin  d'etre  verifiee.  Pour  qu'un 
systeme  multicapteur  puisse  fonctionner  d'une  fagon 
nominale  dans  toutes  les  conditions  pour  lesquelles  il  a 
ete  congu,  il  faut  analyser  le  contexte  et  s'y  adapter.  Le 
resultat  est  simplement  de  privilegier,  dans  les 
algorithmes,  les  mesures  issues  des  capteurs  en  etat 
nominal  de  fonctionnement  et  de  reduire  l'importance  de 
celles  qui  sont  aberrantes  au  vu  de  certains  criteres 
etablis  au  prealable. 

Nous  proposons  dans  cet  article  une  methode,  et  les 
algorithmes  associes,  permettant  de  prendre  en  compte  le 
contexte  pour  un  systeme  multicapteur.  Cette  methode 
elargit  les  travaux  initialement  menes  dans  [8],  [9], 
L'organisation  de  cet  article  est  la  suivante  :  le  principe 
general  de  la  methode  est  presente  dans  la  deuxieme 
partie  ainsi  que  la  logique  de  fonctionnement  qui  en 
resulte.  La  troisieme  partie  decrit  les  equations 
d'estimation  qui  prennent  en  compte  le  contexte.  Nous 
distinguerons  l'estimation  statique,  faite  a  partir  d'un 
ensemble  de  mesures  acquis  a  un  instant  donne,  de 
l'estimation  dynamique,  calculee  a  partir  de  toutes  les 
mesures  passees,  et  qui  conduit  a  des  equations  de 
filtrage  specifiques.  Le  dernier  chapitre  est  consacre  a  une 
simulation. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 


26-2 


2.  Logique  de  fonctionnement  du 
systeme 

2.1  PRINCIPE  GENERAL 


On  considere  qu'un  contexte  particulier  peut  etre  identifie 
par  des  variables  dites  "contextuelles".  La  nature  et 
l'origine  de  ces  variables  sont  tres  diverses.  On  peut 
considerer  des  mesures  faites  par  des  capteurs  annexes 
tels  que  ceux  mesurant  la  pluviometrie  ou  la  temperature 
exterieure,  etc...  De  meme  un  operateur,  par 
l'intermediaire  d’une  interface  homme/machine,  peut 
donner  des  indications  precieuses  sur  les  conditions 
operationnelles  du  moment.  Des  traitements 
supplementaires  mesurant  un  rapport  signal  sur  bruit,  la 
largeur  d'un  pic  de  correlation  ou  tout  autre  parametre  ou 
indicateur  permettant,  dans  certain  cas,  d'evaluer  la 
qualite  du  signal  ou  d'indiquer  l'etat  de  fonctionnement  de 
chaque  capteur,  peuvent  etre  pris  en  compte. 

La  representation  des  connaissances  qui  decrivent  l'etat  de 
fonctionnement  de  chaque  capteur,  et  done  la  qualite  des 
mesures  qu'il  est  susceptible  de  fournir,  est  basee  sur  une 
description  en  termes  de  sous-ensembles  flous.  Cette 
connaissance  est  etablie  par  un  expert  capable  d'evaluer 
les  performances  et  les  limites  de  chaque  capteur,  au 
moyen  de  fonctions  d'appartenance  definies  sur  les 
variables  contextuelles.  Des  lors,  en  situation  reelle,  et 
connaissant  les  valeurs  que  prennent  chaque  variable 
contextuelle,  la  conjonction  des  fonctions  d'appartenance 
etablira  la  validite  des  mesures  issues  de  l'un  ou  l'autre 
des  capteurs.  Ainsi,  il  est  possible  de  definir 
l'association  de  capteurs  la  mieux  adaptee  a  un  contexte 
particulier,  et  de  ne  prendre  en  compte,  dans  le  processus 
d'estimation,  que  les  mesures  issues  de  cette  association. 


2.2  PARTITIONNEMENT  DE  L'ESPACE 
CONTEXTUEL 

La  prise  en  compte  du  contexte  apparalt  comme  une  idee 
assez  naturelle  pour  une  personne  en  charge  de  la 
realisation  d'un  systeme.  Cependant  sa  mise  en  ceuvre 
effective  n'est  pas  immediate  et  aboutit  souvent  a 
^elaboration  de  quelques  heuristiques  et  a  1'evaluation  de 
coefficients  dit  "  de  confiance"  ;  l'ensemble  fournissant 
un  resultat  satisfaisant  quoique  dependant  de  l'application 
concernce.  II  n'existe,  a  notre  connaissance,  pas  de 
methodologie  generale  permettant  de  formaliser  le 
probleme.  Nous  proposons  dans  cet  article  un 
formalisme.  Celui-ci  permet  de  definir  un  espace 
contextuel  et  d'etablir,  sur  cet  espace,  une  logique  de 
fonctionnement  du  systeme.  Cette  logique  sera  ensuite 
utilisee  pour  superviser  le  processus  d'estimation. 


2.2.1  Definition 

Nous  considererons  dans  la  suite  un  systeme  S  constitue 
de  n  capteurs.  Les  variables  contextuelles  seront 
designees  par  les  lettres  zj,  avec  j  E  { I  p  },  p  etant 
le  nombre  de  variables  contextuelles  considerees.  Un 
contexte  particulier  z  est  done  defini  par  p  mesures  ou 
valeurs  de  chaque  variable  contextuelle,  si  bien  que  l’on 
peut  noter  z  =  (zj,  ...  zp}.  Les  contextes  appartiennent  a 

un  espace  a  p  dimensions  note  Z.  Un  capteur  est  valide 
pour  un  ensemble  donne  de  contextes  qui  est  represente 
par  un  sous-ensemble  note  Cj  de  Z,  avec  i  E  {1  ,...,  n 
}.  La  figure  I  illustre  cet  aspect  pour  un  systeme 
compose  de  trois  capteurs.  Chaque  sous-ensemble  Q  , 
avec  i  E  { 1,  2,  3  },  est  represente  ainsi  que  toutes  les 
intersections  entre  les  sous-ensembles. 

z 


Partitionnement  de  iespace  contextuel 
Figure  1 

Nous  distinguerons  ici  deux  domaines  de  validite,  pour 
chaque  capteur,  represents  par  les  deux  notations  Cj  et 
cj.  Le  domaine  de  validite  inclusif  Ci  represente  le 
sous-ensemble  de  contextes  pour  leque!  le  capteur  n°l  est 
valide,  sans  prejuger  toutefois  de  la  validite  des  autres 
capteurs.  Le  domaine  de  validite  exclusif  ci  represente  le 
sous-ensemble  de  contextes  ou  seul  le  capteur  n°l  est 
valide  a  l'exclusion  de  tous  les  autres.  La  relation 

logique  qui  lie  les  deux  est  de  la  forme:  c[=CjnC2nC3. 
Cette  distinction  peut  etre  faite  pour  toute  combinaison 

de  capteurs,  ainsi  c { 1,2}  =  CjnC2nC3  est  le  sous- 
ensemble  de  contextes  dans  lequel  les  capteurs  numero  1 
et  2  sont  valides  mais  pas  le  capteur  numero  3.  Plus 
generalement,  pour  un  systeme  comprenant  n  capteurs, 
on  peut  constituer  l'ensemble  A  =  {  c0,  Cj>  C2,  •••, 
C{1,2}  >  •••>  •••  c { l ,2,  ....  n } }  des  domaines  de  validite 
exclusif  de  toutes  les  combinaisons  de  capteurs,  avec 
cj  =  nCj  n_Ci  et  J  E  {1,  ...,  n}  L'ensemble  A, 
jeJ  ieJ 

constitue  d'elements  exclusifs,  forme  une  partition  de  Z. 
Notons  que  c0  represente  une  absence  de  capteur  valide 

c0  =  Cinc2n  ...  ncn.  Dans  la  suite,  nous  noterons 
cj  un  element  de  A,  ou  J  designe  un  sous-ensemble  de 
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l'ensemble  d'indices  representant  les  capteurs 

valides. 


2.2.2  Remarque 

La  representation  precedente  appelle  la  remarque 
suivante.  L'ensemble  des  contextes  pour  lesquels  les 
trois  capteurs  sont  simultanement  valides  CiflC2nC3 
contient,  a  l'evidence,  un  nombre  de  contextes  moins 
important  que  chaque  sous-ensemble  Cj  pris 
individuellement.  De  ce  fait,  une  stricte  utilisation  d'un 
systeme  multicapteur  aux  conditions  pour  lesquelles 
l'ensemble  des  capteurs  est  valide  limite  drastiquement  le 
champ  d'utilisation  de  ce  systeme.  D'ou  la  necessite 
d'identifier  les  contextes  et  de  considerer  alors 
l'association  de  capteurs  adaptee  a  chacun  d’eux. 


2.3PROBABILITE  de  fonctionnement 
D  UN  CAPTEUR 


2.3.1  Logique  binaire 

En  logique  binaire,  le  sous-ensemble  Cj  de  Z  peut  etre 
represente  par  sa  fonction  indicatrice  d'ensemble  Ij(z). 
Cette  fonction  prend  alors  les  deux  valeurs  1  ou  0 
suivant  que  le  capteur  i  est  valide  ou  ne  Test  pas  pour  le 
contexte  z.  Des  lors  que  z  est  un  vecteur  compose  de  p 
variables  elementaires  Zj,  la  fonction  indicatrice  Ij(z)  est 

obtenue  par  l'operation  logique  de  conjonction  A  des 
fonctions  indicatrices  elementaires  Ijj(Zj). 

Ii(z)  =  Iii(zi)Ali2(z2)  -  Alip(Zp) 

Chaque  fonction  elementaire  Ijj(zj)  definit  la  plage  de 
validite  du  capteur  i  dans  le  contexte  identifie  par  la 
variable  Zy  Par  convention,  on  prendra  Ijj(Zj)=l  si  la 
variable  Zj  ne  renseigne  en  rien  sur  la  validite  du 
capteur  i. 

L'origine  des  variables  z-  conduit  a  considerer  z  comme 
c  J 

un  vecteur  aleatoire  de  densite  de  probabilite  p(z  /  z  )  ou 
zm  est  le  vecteur  representant  les  valeurs  mesurees  de  la 
variable  z.  Cette  densite  modelise  l'incertitude  liee  a  la 
mesure.  La  prise  en  compte  de  cette  incertitude  ne 
permet  plus  d'etablir  d'une  faqon  binaire  la  validite  d'un 
capteur  mais  d'evaluer  la  probabilite  pour  que  le  capteur 
soit  valide.  Celle-ci  est  donnee  par  la  formule  : 

P(C;  /  zm)  =  J  Ij  (z)  p(z  /  zm)  dz 

Connaissant  la  valeur  des  parametres  mcsures  zm, 
P(Ci/zm)  represente  la  probabilite  pour  que  la  valeur  de  z 
appartienne  au  sous-ensemble  Cj . 


2.3.2  Logique  floue 

La  logique  tloue  apporte  une  nuance  supplementaire  dans 
la  representation  des  plages  de  validite  de  chaque  capteur. 
En  effet,  dans  une  logique  binaire,  les  bornes  definissant 
ces  plages  sont  souvent  arbitraires,  elles  peuvent  etre 
avantageusement  remplacees  par  des  intervalles  sur 
lesquels  la  validite  de  chaque  capteur  est  mesuree  par  une 
variable  prenant  ses  valeurs  dans  l'intervalle  [0,1].  Les 
fonctions  indicatrices  sont  alors  les  fonctions 
d'appartenance  |ij(z)  du  sous-ensemble  flou  Cj,  la 
notation  restant  ici  identique  a  celle  adoptee  en  logique 
binaire  puisque  aucune  confusion  ne  peut  etre  faite. 
Comme  precedemment  |ij(z)  est  obtenue  par  conjonction 
des  fonctions  d'appartenance  elementaires  j-iy(Zj). 

W(z)  =  Hii(z1)Ajii2(z2)  ...  Anip(zp) 

ou  A  est  un  operateur  de  conjonction  de  logique  floue. 
Comme  precedemment,  des  lors  qu'un  capteur  i  n'a  pas 
de  lien  direct  avec  un  contexte  identifie  par  la  variable  Zj, 
la  convention  est  de  prendre  Hi(zj)  =1  pour  toutes  les 
valeurs  que  prend  zj. 

L'utilisation  de  fonction  d'appartenance  de  sous-ensemble 
flou  permet  de  s'affranchir  d'un  arbitraire  portant  sur  la 
definition  des  bornes  de  validite  de  chaque  capteur.  Ainsi 
elle  donne  la  possibility  de  modeliser  l’incertitude 
inherente  a  cette  definition.  Une  seconde  source 
d'incertitude  reside  toujours  dans  la  composante  aleatoire 
des  variables  Zj.  La  combinaison  des  deux  incertitudes 
est  a  l'origine  de  la  definition  de  la  probabilite  d'un 
evenement  flou  proposee  par  Zadeh  [14]  suivant  la 
relation  : 

P(C;  /  zm)  =  J  m  (z)  p(z  /  zm)  dz  (1) 

Lorsque  l'incertitude  liee  a  la  mesure  est  negligeable,  ou 
encore,  si  la  valeur  de  la  variable  est  certaine,  la  densite 
de  probabilite  p(z  /  zm)  est  remplacee  par  un  dirac 
8(z-zm)  permettant  l'identification  de  la  probabilite  de 
l’evenement  flou  a  la  valeur  que  prend  la  fonction 
d'appartenance  au  point  considere.  P(Cj/zm)  represente  la 
probabilite  pour  que  la  valeur  de  z  appartienne  au  sous- 
ensemble  flou  Cj  sachant  que  le  contexte  mesure  est 
zm. 

Lorsque  les  variables  Zj  sont  independantes,  et  en  prenant 
l'operateur  Min  comme  operateur  de  conjonction,  la 
formule  (1)  se  developpe  suivant : 

P(Cj  /  zm)  =  J  Min  (ftj l (z j ),...,  |ijp  (zp))  pCzi/z^) 

...  p(zp/Zp)dzj  ...  dzp 
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2.4  PROBABILITIES  D'UN  GROUPEMENT 
DE  CAPTEURS 


2.4. 1  Probabilite  d' association 

Plusieurs  capteurs  peuvent  etre  associes,  la  probabilite 
du  groupement  qui  en  resulte  est  egale  a  la  probabilite  de 
la  conjonction  des  evenements  flous  associes  a  chaque 
capteur.  Pour  deux  evenements  cette  probabilite  est 
definie  par  la  relation  suivante  : 

P(Cj  nCj  /  zm)  =  |  Min  (|J.j(z),  fij(z) )  p(z/zm)  dz, 

lorsque  Min  est  pris  corame  operateur  de  conjonction. 
La  generalisation  a  l'intersection  de  plusieurs 
evenements  est  immediate. 


2.4.2  Probabilite  du  domaine  de  validite  exclusif 

La  probabilite  d'un  domaine  de  validite  exclusif  en 
fonction  des  domaines  de  validite  inclusifs  est  definie  par 
la  formule  suivante,  dont  la  demonstration  est  donnee  en 
annexe  : 

Pj=P(cj)  =  S  (-i)II'JIP(ni£lci)  (2) 

(1C  N/JSl  ] 


et  Pp  -  Pfcp)  -  P(njENCj) 

ou  I  et  J  sont  deux  ensembles  d'indices  correspondant 
chacun  a  une  partie  de  l'ensemble  N={  1,  „n).  On  note  II- 
Jl  le  cardinal  de  l'ensemble  I-J.  Pour  des  raisons  de 
simplicite  d'ecriture,  nous  avons  omis  d'ecrire  le 
conditionnement  par  la  variable  mesuree  zm  dans  les 
probability.  Ainsi,  P(cj)  =  P(cj/zm)  represente  la 

probabilite  pour  que  le  contexte  mesure  zm  appartienne 
au  domaine  de  validite  exclusif  cj.  II  existe  alors  autant 
de  probabilite  P(cj)  qu'il  y  a  d'elements  dans  A  c'est-a- 
dire  2n.  La  condition  de  normalisation  etant  verifiee  : 

X  p,  =  i 

JCNU0 

3.  Estimation  avec  prise  en 
compte  du  contexte. 

Les  probability  qui  viennent  d'etre  definies  ci-dessus 
permettent  d'etablir  la  validite  de  chaque  groupement  de 
capteurs.  Ainsi,  elles  permettent  de  valider  les  mesures 
issues  des  differents  capteurs  qui  composent  le 
groupement  et,  de  ce  fait,  l'estimee  fournie  par  la  fusion 
partielle  de  celles-ci.  Lorsque  plusieurs  groupements 
sont  simultanement  valides,  avec  des  probability 
differentes,  la  fusion  globale  est  le  resultat  de  la 


moyenne  des  fusions  partielles  ponderees  par  les 
probability  qui  leur  sont  associees. 

Nous  distinguerons  alors  deux  cas  suivant  que  1'etat  ne 
depend  que  de  l’instant  present  (  estimation  statique  )  ou 
bien  des  etats  passes  (estimation  dynamique).  Les 
applications  visees  different  d'un  cas  a  l'autre. 


3.1  ESTIMATION  STATIQUE 

3.1.1  Formulation  du  probleme 

Le  systeme  est  compose  de  n  capteurs  chacun  delivrant 
une  mesure  y\  avec  i£  N,  permettant  l’observation  d'un 
etat  x  a  travers  n  equations  d'observation  : 

y‘  =  ^  (x,  b1) 

ou  Hj,  i  E  N,  sont  les  n  systemes  d'observation,  b1  les 
bruits  d'observation.  Par  souci  de  simplicite  nous 
considererons  que  le  bruit  est  additif,  gaussien,  de 

moyenne  nulle  et  de  variance  E(bV)  =  af  8(i,j)  ou  5  est 
le  symbole  de  Kronecker.  L'ensemble  des  observations 
disponibles  est  regroupe  dans  le  vecteur  d'observation 
YT=  {  y1  yn). 

3.1.2  Equations  d'estimation 

L'estimation  de  l'etat  x,  optimal  en  moyenne 
quadratique,  est  obtenue  par  la  moyenne  conditionnee 
aux  observations  de  la  variable  x  : 

x  =  E(x/Y)  =|  x  p(x  /  Y)  dx  (3) 

On  peut  developper  la  probabilite  p(x  /  Y)  sous  la  forme 
suivante  : 

P(x  1  Y)  =  X  P(x  /  Y,  cj)  P(cj)  (4) 

JCNU0 

II  existe  done  2n  probabilites  elementaires  p(x  /  Y,cj) 
affectees  a  x  et  correspondant  aux  parties  de  N. 

L'estimation  de  x  avec  prise  en  compte  du  contexte 
s'obtient  en  remplagant  la  probabilite  dans  (3)  par 
l'expression  donnee  en  (4) : 

x— P(c0> J  xp(x/Y,Crt)dx+~X  P(cj) J*  xp(x/Y,cj)dx  (5) 

J  S  N 

Le  conditionnement  par  cj  signifie  que  seuls  les  capteurs 
dont  les  indices  sont  contenus  dans  J  sont  valides.  De  ce 
fait,  seules  les  observations  correspondant  a  ces  capteurs 
doivent  etre  considerees.  Ces  observations  sont  alors 
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regroupees  dans  le  vecteur  Yj  =  {ykJksJ  ce  I0*  Permet 
d'ecrire  la  relation  (5)  sous  la  forme  : 

x  =  xo(30  +  X  Pj  E(x  1  Yj)  (6) 

JCN 


E(x  /  {yi })  =  y i 
E(x/  {y2))  =  y2 

°2  al 

E(x  /  { y l ,y2) )  =  Y\  2 - r+y2~2 - T  W 

O’ |  +0  9  Oj  +O2 


Les  coefficients  (3j  sont  donnes  par  la  relation  (2). 

Lorsqu'aucun  capteur  n'est  valide,  on  pose  : 

La  variable  xo  qui  apparait  dans  (6)  correspond  a  la 

quantite  :  E(x/{0})=  xq 


xo  =  J  x  p(x  /  Y,  c0  )  dx 

Comme  c0  represente  l'absence  de  capteur  valide,  xo  est 
done  fixee  a  priori  et  doit  etre  consideree  comme  la 
valeur  limite,  par  defaut,  que  doit  prendre  l'etat  si  aucun 
capteur  n’est  en  mesure  de  delivrer  une  observation 
coherente. 

Dans  certains  cas  cette  situation  peut  paraitre  absurde. 
On  suppose  alors,  qu'a  tout  instant,  un  des  capteurs  au 
moins  est  en  mesure  de  delivrer  une  donnee  acceptable, 
l'equation  (6)  peut  etre  mise  sous  la  forme  : 


oil  xo  est  une  donnee  a  priori. 

L'estimateur  globale  est  fourni  par  la  formule  (6)  ou  l'on 
a  remplace  les  probabilites  Pj  par  leurs  valeurs  donnees 
en  (8),  et  les  differentes  estimees  par  leurs  valeurs 
donnees  en  (9).  Le  resultat  donne  alors  : 

a  a\ 

x  =  P(c0)xo+yi[P(Ci)--  2  ,  P(CinC2)]  + 

°1  +<72 

cf 

y2[P(C2)  -  2  2"P(Cinc2)] 

ai  +a2 


X  =  [  X  PjE(x/ Yj)]/[l-p0]  (7) 

JCN 

Chaque  probabilite  Pj  est  divisee  par  1-P0  de  sorte  que  la 
condition  de  normalisation  soit  respectde. 

3.1.3  Exemple 

L'exemple  suivant  permet  d’illustrer  les  equations 
proposees.  Prenons  un  systeme  compose  de  deux 
capteurs  ;  par  exemple  :  un  capteur  radar  Cl  et  une 
camera  visible  C2.  Deux  variables  contextuelles 
permettent  d'identifier  le  contexte  :  z\  est  un  indicateur 
de  fonctionnement  du  radar,  Z2  est  un  parametre  qui 
mesure  la  luminosite.  Les  deux  fonctions 
d'appartenances  qui  permettent  d'etablir  la  validite  de 
chaque  capteur  seront  notees  Pi(zi)  et  M-2(z2)- 

Pour  ce  systeme  la  formule  (2)  fournit  les  probabilites 
suivantes  : 

P  { 0 }  =  P(  c  j  n  C2)  =  P(c0) 

P{1}  =P(Ci)  -P(Cinc2) 

P{2}  =P(C2)  -p(Cinc2) 

P{  1,2}  =P(Cinc2)  (8) 

Si  les  deux  observations  yj  et  y0  sont  independantes, 

aaussiennes,  de  moyenne  x,  et  de  variance 
•  2  2 

respectivement  a  ^  et  a2  ,  pour  chaque  association  de 
capteurs,  les  differentes  estimees  sont : 


Pour  un  operateur  de  conjonction  Min,  et  en  supposant 
que  les  erreurs  sur  les  mesures  de  z\  et  de  z2  sont 
negligeables,  les  probabilites  sont  alors  : 


P(c0)  =  Min(l-m(zi)  ;  l-p.2(z2)) 
p(Ci)  =  m(zi) 

P(C2)  -  fi2(z2) 

P(Cif1C2)  =  Min(lli(zi)  ;  112(2)) 


On  peut  ainsi  etudier  quelques  situations  particulieres  : 

Cas  1  -  Pour  lti(zi)=lt2(z2)=:E  le  contexte  est  favorable 
a  l’utilisation  simultanee  des  deux  capteurs,  l'estimee  est 
alors  : 


a, 


=yi 


Cl  +C2 


y2 


1  +a2 


Cette  estimee  est  aussi  obtenue  par  la  moyenne 
conditionnee  aux  deux  observations  yt  et  y2  et 
correspond  a  une  approche  probabiliste  de  la  fusion  sans 
que  le  contexte  soit  pris  en  consideration. 

Cas  2  -  Pour  M- 1  (zi )=0  et  M-2(z2)=l »  ces  conditions 
invalident  le  capteur  1  alors  que  le  capteur  2  est 
totalement  valide,  on  obtient : 

A 

x  =  y2  . 


Seule  l'observation  du  second  capteur  est  prise  en 
compte.  Ce  resultat  est  conforme  au  souhait  de  voir  le 
systeme  ne  prendre  en  compte  que  les  mesures  issues  de 
capteurs  valides  dans  le  contexte  considere. 
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Cas  3  -  Une  situation  intermediaire  est  donnee  par 
M-l(zl  )=0.5  et  JJ-2(Z2)=  1 »  qui  invalident  partiellement  le 
capteur  1.  L'estimee  est  alors  : 


x  =  yi 


0.5a^ 


a,  +a. 


*2 


a2  +  0.5a j 

2  2 
Oj  +a2 


L’ensemble  des  mesures  fournies  par  le  capteur  i  jusqu'a 
l’instant  k  est  note  Yk  =  {  y{  et  l'ensemble  de 
toutes  les  mesures  pour  tous  les  capteurs  a  l’instant  k 
sera  note  Yk  =  {  Yk  .  De  plus,  pour  toute  partie 
J£N  on  notera  Yk  ={  Y^  }j£j  l'ensemble  des  mesures  a 

l'instant  k  fournies  par  l'association  de  capteurs  identifies 
par  J. 


On  constate  une  ponderation  plus  importante  sur 
l'observation  fournie  par  le  capteur  2  par  rapport  a  celle 
obtenue  dans  le  cas  1  qui  supposait  alors  les  deux 
capteurs  valides. 

Cas  4  -  Pour  M-i(zi)  =  0  et  |l2(z2)=0-  Ce  cas  limite 
correspond  au  cas  ou  aucun  des  capteurs  n'est  valide.  La 
seule  estimee  possible  est  celle  fournie  par  la  valeur  a 
priori  xq. 


3.2.2  Equations  de  filtrage  pour  une  association  de 
capteurs  donnee 

Pour  une  association  de  capteurs  dont  les  indices  sont 
elements  de  J,  l'estimee  au  sens  de  l'erreur  quadratique 
moyenne  est  donnee  par  la  moyenne  conditionnee  : 

xk/k=  E(xk  /  Yk  ) 


3.1.4  Remarque 

L’estimee  E(x  /  { y i  ,y 2 } )  de  la  formule  (9)  que  Ton 
retrouve  aussi  dans  le  cas  1,  est  l'estimee  fusionnee  au 
sens  probabiliste.  On  remarque  que  l'importance  de 
chaque  capteur  dans  le  processus  de  fusion,  qui  n'est  en 
faite  qu'une  simple  moyenne  ponderee,  depend  de  la 
variance  du  bruit  d'observation.  Plus  la  variance  d'un 
capteur  est  forte,  ce  qui  denote  une  qualite  moindre  de  la 
mesure,  plus  la  valeur  du  coefficient  qui  pondere  le 
second  capteur  est  importante.  Ce  processus  permet  de 
privilegier  dans  la  moyenne  ponderee  la  mesure  issue  du 
capteur  de  plus  faible  variance. 

3.2  ESTIMATION  DYNAMIQUE 

3.2.1  Formulation  du  probleme 

On  note  xk  le  vecteur  d'etat  a  l'instant  k.  Le  modele 
representant  la  dynamique  du  systeme  est  suppose  ici 
lineaire,  invariant,  et  d'equation  : 


Les  equations  qui  resultent  de  cette  estimation  sont  les 
equations  de  Kalman  adaptees  a  la  fusion  des  donnees 
[4],  [5]  qui  prend  en  compte  un  nombre  d'equations 
d'observation  supericur  a  1.  L'estimee  optimale  a 
l'instant  k  est  done  fournie  par  la  relation  : 

Xk/k  =  Xk/k-l  +X  K  j(k)  (  yi  -  Hj  xk/ki)  (10) 
j£J 

oil  K^j(k)  est  le  gain  de  Kalman  associe  au  capteur  j 
appartenant  au  groupement  J.  L'ecriture  des  equations  de 
Kalman  pour  un  systeme  multicapteur  est  plus  simple 
sous  la  forme  information,  et  e'est  cette  forme  que  nous 
reprenons  ici,  sachant  que  la  mise  en  ceuvre  effective  du 
filtre  se  prete  mieux  a  une  programmation  sequentielle 
que  nous  presenterons  dans  la  partie  algorithmic.  Le  gain 
de  Kalman  prend  done  la  forme  : 

K-j(k)  =  Pj(k/k)Hj  R'/  (11) 


xk  =  F  xk-i  +  Vk 

F  est  la  matrice  de  transition  du  systeme.  On  suppose 
que  vk  est  un  processus  aleatoire  gaussien  de  moyenne 

T 

nulle  et  de  matrice  de  covariance  :  E(vkVj )  =  Q  5(k,j). 
On  suppose  que  les  n  equations  d'observation  sont 
lineaires,  de  la  forme  : 


La  matrice  des  erreurs  de  predictions  a  posteriori  Pj(k/k) 
est  obtenue  par  la  forme  recurrente  sur  les  inverses 
suivante  : 

Pj(k/k)  =  Pj(k/k-l)  +  X  Hj  R-/  Hj  (12) 
jeJ 


yk  =  H;  xk  +  bk 

aveci£N.  Hj  Hn  sont  les  n  matrices  lignes 
d'observation.  Lorsque  les  equations  d'observation  sont 
non  lineaires  ces  matrices  seront  remplacees  par  les 
Hessiens  ;  les  equations  d'estimation  resultantes  seront 
alors  celles  du  filtre  de  Kalman  etendu.  Les  bruits 
d'observations  bk  ,  ...,  bk  sont  gaussiens  de  moyenne 
nulle  et  variance  donnee  par  E(bkb])  =  Rj  5(k,l,j). 


Pour  un  systeme  compose  de  n  capteur  le  filtre  de 
Kalman  classique  pour  la  fusion  est  base  sur  les 
equations  precedentes  en  remplapant  J  par  N. 

3.2.3  Equations  du  filtre  avec  prise  en  compte  du 
contexte 

Nous  allons  considerer  maintenant  toutes  les 
associations  de  capteurs  J  possibles  avec  J£N  et 
composer  a  partir  de  celles-ci  une  estimee  globale.  D'une 
fapon  generale,  celle-ci  s'ecrit : 
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*k/k-  E(xk  /  Yk) 

La  prise  en  compte  du  contexte  s'effectue,  comme  dans 
le  cas  statique,  par  Ie  developpement  de  l'estimateur  xk/k 
suivant  la  formule  : 

xk/k=  x0  Pp(k)  +  X  Pj  (k)E(xk  /  Yk  )  (13) 

JC  N 


destine  a  la  fusion  de  donnees.  D’une  part,  cette  forme 
est  plus  facile  a  mettre  en  ceuvre,  et  d'autre  part,  elle 
permet  une  extension  immediate  aux  traitement  de 
capteurs  asynchrones.  L'algorithme  est  presente  sous  les 
deux  formes  :  sans  prise  en  compte  du  contexte  et  avec 
prise  en  compte  du  contexte.  Nous  considererons,  pour 
les  besoins  de  la  presentation,  un  systeme  compose  de 
trois  capteurs,  mais  la  generalisation  a  un  systeme 
compose  de  n  capteurs  ne  pose  aucune  difficulte. 


L'estimee  correspondant  a  la  fusion  globale  est 
decomposee  en  fonctions  des  fusions  partielles.  En 
1'absence  de  mesure  valide,  xO  peut  etre  avantageusement 
remplace  par  la  prediction  de  l'etat  xk/k_i.  Les  fusions 
partielles  E(xk  /  Yk  )  sont  remplacees  par  leurs  valeurs 
donnees  en  (10),  oil  l'on  a  substitue  chaque  prediction 
au  sens  de  1'association  J  par  la  prediction  de  l'etat 
Xk/k- 1  fournie  par  la  fusion  globale  a  l'instant  precedent. 
La  relation  (13)  devient : 

xk/k=Xk/k-l+X  X  Pj(k)  Ki(k)(yi-  Hj4/k.i)](14) 

JCN  jeJ 

Cette  derniere  relation  peut  prendre  une  forme  plus 
simple  en  adoptant  la  notation  : 


3.2.4. 1  Algorithmes  de  fusion  sans  prise  en  compte  du 
contexte. 

Nous  rappelons  ici  l'algorithme  sequentiel  du  filtre  de 
Kalman  pour  un  systeme  multicapteur  synchrone  [5]. 
Les  equations  d'etat  pour  un  systeme  S  compose  de  trois 
capteurs  sont  les  suivantes  : 

f  xk  =  Fxk.i+vk 

s  <!  yk=  Hi  xk  +  bk 

j  yk  =  H2  Xk  +  bk 

l  yk  =  H3  Xk  +  bk 


Dynamique 

Capteur  1 
Capteur  2 
Capteur  3 


Une  forme  equivalante  de  ce  systeme  d'equation  est  la 
Kj  (k)  =  2-i  Pj  (k)KJj(k)  (15)  suivante  : 

{J/JEJ} 


oil  {J  /jEJ}  est  l'ensemble  de  tous  les  groupements  de 
capteurs  qui  contiennent  le  capteur  j,  ce  qui  conduit  a  : 

xk/k=  xk/k-l  +  X  Kj  (R)  (  yi  -  Hj  xk/k-l)]  (16) 
jeN 

La  matrice  de  covariance  a  posteriori  est  obtenue  grace  a 
la  relation  suivante  [2]  : 

£k/k=  X  Pj(k)[^j(k/k)+(xk/k-xk/k)(xk/k-xk/k)T] 
J£NU0 

(17) 


xk=  F  xk.i  +vk 
yk=  Hi  xk  +  bk 


Dynamique 
Capteur  1 


S2  \ 


2  1 
xk  —  xk 

2  u  2  ,  ,2 

yk  -  H2  xk  +bk 


Dynamique 
Capteur  2 


S3  \ 


x  k=  xk  Dynamique 

3  3 

yk  -  H2  xk  +  bk  Capteur  3 


qui  correspond  a  la  matrice  de  covariance  d'un  melange  de 
lois  gaussiennes.  Pj(k/k)  est  la  matrice  de  covariance  a 
posteriori  correspondant  a  chaque  groupement  de  capteurs 
J. 

Enfin,  a  partir  de  l'etat  et  de  la  matrice  de  covariance,  les 
equations  de  prediction  sont : 

xk+l/k  -  F  x^  et  ^k+i/k  =  F^k/kFT  +  Q  (18) 
3.2.4  Algorithmes 


Cette  decomposition  suggere  done  d'effectuer  le 
traitement  en  quatre  etapes  successives  :  une  etape  est 
consacree  a  la  prediction  les  trois  autres  sont  des 
processus  d'estimation.  Le  premier  systeme  d'equations 
note  SI  est  a  1'origine  des  deux  premieres  etapes  :  la 
prediction  suivi  du  renouvellement  de  l'etat  par  la  donnee 
issue  du  capteur  1.  Les  deux  etapes  suivantes  sont 
consacrees  au  renouvellement  de  l'etat  respectivement  par 
la  donnee  du  capteur  2  et  celle  du  capteur  3. 


Bien  que  les  equations  precedentes  puissent  etre 
directement  utilisees,  on  prefere  souvent  la  forme 
sequentielle  [5]  pour  l'implantation  d’un  filtre  de  Kalman 
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Etape  1 

A  partir  d'un  etat  estime  a  1'iteration  k-1,  la  prediction 
s'effectue  de  faqon  classique  par  l'application  des 
formules  (18)  ci-dessus. 

Etape  2 

La  prediction  est  ensuite  mise  a  jour  par  les  formules 
classiques  suivantes  : 

*k/k  =  xk/k-1  +  Kj(k)  (  yk  -  Hi  Xk/k-l) 
et 

Pkk/k)  =  (I  -  Kj(k)  Hi)Pk/k-i  (19) 

avec  : 

Kj(k)  =  Pk/k-1  H|(Hi  Pk/k-iHl+Ri)-1 
Etape  3 

Au  vu  du  systeme  S2,  le  processus  de  prediction  est 
simplement  obtenu  en  identifiant  l'etat  predit  par  l'etat 
estime  a  l'etape  precedente,  et  la  matrice  de  covariance 
des  erreurs  a  priori  par  la  matrice  des  erreurs  a  posteriori 
obtenue  a  l'etape  precedente.  L'etape  d'estimation  est 
obtenue  en  substituant  dans  les  equations  (19)  Pk/k-1 

par  p!(k/k),  xk/k-l  Par  %/k  et  en  remplaqant  l’indice 
1  par  l'indice  2  sur  les  autres  quantites. 

Etape  4 

Cette  quatrieme  etape  est  rigoureusement  identique  a 
l'etape  precedente.  A  la  suite  de  cette  etape  il  suffit  se 
changer  pl(k/k)  par  P2(k/k)  et  xj?/^  par  x^/k  et  de 
remplacer  l'indice  2  par  l'indice  3  sur  les  autres  quantites. 
L'estimee  ainsi  que  la  matrice  de  covariance  des  erreurs  a 
posteriori  issues  de  cette  etape  sont  maintenant  l'estimee 
et  la  matrice  de  covariance  des  erreurs  a  posteriori  du 
systeme  global  S  et  sont  notees  xk/k  et  Pk/k-1- 

Dans  la  simulation  qui  va  suivre,  les  resultats  obtenus 
par  l'algorithme  qui  vient  d'etre  presente  seront  notes  xf. 

Remarque 

L'algorithme  precedent  donne  des  resultats 
rigoureusement  identique  a  ceux  resultant  de  la 
programmation  des  equations  du  filtre  de  Kalman  sous  la 
forme  information  presentees  section  3.2.2.  Par  ailleurs, 
et  en  faisant  le  parallele  a  la  remarque  3.1.4,  on  remarque 
qu'a  chaque  etape  la  variance  du  bruit  d'observation  du 
capteur  considere  est  prise  en  compte  dans  le  gain  de 
Kalman.  Cette  procedure  permet  de  tenir  compte  de 
l'importance  des  capteurs  dans  le  processus  de  fusion.  En 
effet,  la  matrice  de  covariance  du  bruit  d'observation  est 
differente  d'un  capteur  a  l'autre,  celle-ci  depend 


notamment  de  la  resolution  du  capteur  et  du  rapport 
signal  sur  bruit.  Plus  cette  matrice  est  importante,  au 
sens  d'un  critere  tel  que  la  trace,  moins  les  mesures  du 
capteur  correspondant  auront  de  l'importance  dans  le 
processus  global  de  fusion.  Cependant,  la  hierarchisation 
des  capteurs  en  fonction  d'un  seul  critere  statistique  n'est 
pas  suffisante  pour  apprehender  le  processus  de  fusion 
dans  sa  globalite. 


3. 2. 4. 2  Algorithmes  de  fusion  avec  prise  en  compte  du 
contexte. 


L'algorithme  propose  avec  prise  en  compte  du  contexte 
est  base  sur  l'algorithme  precedent,  avec  cette  difference 
que  Ton  doit,  pour  un  systeme  multicapteur  comportant 
n  capteurs,  determiner  2n-l  estimees,  au  lieu  des  n 
estimees  dans  la  version  precedente.  Chaque  estimee 
correspond  a  une  association  de  capteurs.  La 
combinatoire  n'est  pas  ici  un  obstacle  car,  dans  les 
systemes  actuels,  n  est  de  l'ordre  de  deux  ou  trois  pour 
chaque  coordonnee  (site,  azimut,  gisement). 

Nous  distinguerons  ici  5  etapes  dans  le  traitement.  La 
premiere  etape  est  toujours  dediee  a  la  prediction.  Les 
trois  suivantes  se  distinguent  par  le  nombre  de  capteurs 
inclus  dans  les  differentes  associations.  La  cinquieme 
etape  est  celle  qui  permet  de  prendre  en  compte  le 
contexte  en  attribuant  a  chaque  association  une 
probability,  et  en  effectuant  la  moyenne  ponderee  des 
differentes  estimees. 

Etape  1 

Comme  precedemment  cette  etape  effectue  la  prediction. 
A  partir  des  estimees  acquises  a  1'iteration  precedente  Xk- 
1/k-l  et  Pk-l/k-1  elle  permet  d'etablir  les  predictions 
Xk/k-1  et  Pk/k-1  a  partir  des  equations  (18). 

Etape  2 

Cette  deuxieme  etape  consiste  a  estimer,  pour  chaque 
capteur  i,  l'etat ^k/k  et  matrice  de  covariance  P^/k  a 
partir  de  chaque  donnee  y£  avec  i  E  { 1,2,3}.  Chaque 
vecteur  d'etat  et  sa  matrice  de  covariance  seront 
consideres  ensuite  comme  des  donnes  a  priori  dans  la 
seconde  phase  d'estimation. 

Etape  3  . 

On  realise  ensuite  la  fusion  partielle  de  toutes  les  paires 
de  capteurs  qu'il  est  possible  de  former  dans  le  systeme. 
Dans  le  cas  present  on  en  denombre  trois  qui  sont  : 
{ 1,2},  {2,3},  {3,1 }.  L'estimee,  que  nous  noterons  Xk"/k- 
et  sa  matrice  de  covariance  Pk~/k-  seront  alors  calculees 
en  prenant  comme  etat  a  priori  Xk/k  et  ?k/k  et  en 
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mettant  a  jour  cet  etat  grace  a  la  donnee  yjj .  Cette  mise 
a  jour  tient  compte  des  caracteristiques  du  capteur  2 
notamment  dans  le  calcul  du  gain  et  de  la  matrice  de 
covariance.  Ce  processus  est  reitere  pour  toutes  les 
paires  de  capteurs  consideres.  La  convention  etant  que  le 
premier  indice  designe  l'etat  a  priori  et  le  second  designe 
le  capteur  effectuant  la  mise  a  jour. 

Etape  4 

Cette  troisieme  etape  est  celle  qui  effectue  la  fusion 
globale  des  trois  capteurs.  L'etat  a  priori  est  alors  1'un 
des  etats  resultant  de  l'une  des  fusions  partielles  des 
paires  precedentes.  L’etat  est  ensuite  mis  a  jour  par  la 
mesure  du  capteur  dont  l'indice  n’est  pas  contenu  dans  la 
paire  a  priori.  Par  exemple  la  paire  a  priori  peut  etre  la 
paire  {1,2}  donnant  l'etat  xj^/k  et  sa  matrice  de 

covariance  P  k  /k  .  la  mise  a  jour  est  alors  effectuee  par 

a1  23 

la  mesure  issue  du  capteur  3  pour  obtenir  x  k  /k  et  sa 

matrice  de  covariance  ^  {p/k  •  H  apparait  evident  que 
l'ordre  des  indice  n'a  pas  d'importance  et  que  l'etat 
designe  par  iPjp/k  es^  identique  a  celui  designe  par 

^kPk-  L'etat  estime  x^/k  etant  celui  qui  serait 
obtenu  si  le  contexte  n'etait  pas  pris  en  compte,  il  est 
done  identique  a  celui  obtenu  par  l'algorithme  precedent 
si  Ton  considere  que  les  etats  a  priori  sont  les  memes. 

Etape  5 

Cette  quatrieme  phase  permet  de  prendre  en  compte  le 
contexte.  A  l'instant  de  l'estimation  chaque  variable 
contextuelle  a  une  valeur  determinee  qui  permet  d'evaluer 
la  valeur  de  la  fonction  d'appartenance  correspondante  et 
ensuite  d'en  deduire  l'ensemble  des  coefficients  |3j(k). 

Ces  coefficients  vont  permettre  d'evaluer  l'etat  global 
Xk/k  a  partir  de  l'equation  (10)  ainsi  que  la  matrice  de 
covariance  Pk/k  a  partir  de  l'equation  (17). 

Dans  la  simulation  qui  va  suivre,  les  resultats  obtenus 
par  l’algorithme  qui  vient  d'etre  presente  seront  notes  xc. 

Remarque 

En  plus  du  critere  statistique,  un  second  critere  intervient 
dans  l'attribution  de  l'importance  de  chaque  capteur  dans 
le  processus  de  fusion.  Ce  second  critere  resulte  d'une 
analyse  contextuelle  et  mesure  l'adequation  du  capteur  au 
contexte. 


4.  Simulation 


4.1  Conditions  de  simulation 

La  simulation  presentee  ici  a  pour  objet  la  fusion  de 
mesures  issues  d'un  systeme  compose  de  trois  capteurs  : 


un  radar  de  veille  represente  par  la  donnee  y3,  un  radar  de 
poursuite  represente  par  la  donnee  y?,  et  une  camera 
infrarouge  represente  par  la  donnee  yj.  Deux  des  capteurs 
subissent  des  perturbations.  Les  mesures  yj  et  y2  issues 
des  capteurs  1  et  2  ont,  dans  les  conditions  normales,  les 
memes  caracteristiques  de  bruit :  celui-ci  est  gaussien, 
de  moyenne  nulle,  et  d'ecart  type  dj  =  G2  =  L  La 
mesure  issue  du  capteur  3  a  un  bruit  gaussien,  de 
moyenne  nulle,  et  d'ecart  type  (T3  =  5.  Pour  les  capteurs 
1  et  2  des  perturbations  accidentelles  se  rajoutent  au 
bruit  de  mesure.  Pour  le  capteur  2  le  phenomene 
accidentel  est  un  bruit  blanc  gaussien,  se  rajoutant  au 
premier,  de  moyenne  nulle  et  d'ecart  type  10.  Par  ailleurs 
on  suppose  connu,  par  la  variable  contextuelle  Z2(t), 
1'evolution  de  cette  perturbation  en  fonction  du  temps  ; 
celle-ci  est  donnee  figure  2.  Le  second  phenomene, 
attache  au  capteur  1,  est  de  type  impulsionnel.  La 
probability  d'apparition  d’une  impulsion  a  l'instant  t  : 
I(t)  est  donnee  par  une  loi  binomiale  de  probability  0.15 
et  l'amplitude  de  la  perturbation  en  fonction  du  temps  est 
donnee  par  la  formule  : 

pt(t)=zi(t)W  x2 


ou  x  est  une  variable  aleatoire  gaussienne  de  moyenne 
nulle  et  d'ecart  type  egal  a  10.  La  variable  contextuelle 
zj(t)  indiquant  la  presence  d'une  perturbation  est  donnee 
figure  2. 


Figure  2 

Variables  contextuelles 

Z2  est  la  variable  contextuelle  indiquant  une 
perturbation  sur  le  capteur  2.  z]  est  la  variable 
contextuelle  indiquant  une  perturbation  sur  le  capteur  1. 
Ces  variables  representent  la  presomption  qu'une 
perturbation  affecte  I’un  des  capteurs  mais  dont  les 
caracteristiques  sont  inconnues. 
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Les  signaux  resultant  de  cette  simulation  sont  presentes 
figure  3. 


Figure  3 

Signaux  simules. 
yy  :  Radar  de  veille 
y2 :  Radar  de  poursuite 
y I  :  Camera  I.R. 

4.2  Resultats 

Les  resultats  des  estimations  sont  presentes,  figure  4, 
dans  le  cas  d'une  fusion  sans  prise  en  compte  du 
contexte,  xf,  et  dans  le  cas  d'une  fusion  avec  prise  en 
compte  du  contexte,  notee  xc.  L'estimee  xf  est  calculee 
par  la  methode  sequentielle  telle  qu'elle  est  presentee 
dans  [5]. 


Figure  4 
Resultats 

xf  est  le  resultat  du  filtrage  par  un  filtre  de  Kalman 
adapte  a  la  fusion  de  donnees  mais  sans  exploiter 
I'information  apportee  par  les  variables  contextuelles  de 
la  figure  2. 

xc  est  le  resultat  obtenu  par  le  filtre  propose  qui  tient 
compte  des  variables  contextuelles  de  la  figure  2. 


On  remarque  que  les  perturbations  affectent  assez  peu  xc 
comparativement  a  xf.  Ce  constat  est  confirme  au  vu  des 
courbes  d'erreur  presentees  figure  5.  Aux  instants  ou  il 
n’existe  aucune  perturbation  les  deux  courbes  sont 
identiques  comme  c'est  le  cas  au  debut  et  a  la  fin  de  la 
simulation  ;  ailleur  l'erreur  attachee  a  xc  est  presque 
toujours  inferieure  a  celle  associee  a  xf. 


Figure  5 

Erreur  quadratique  moyenne. 

_ :  erreur  de  xc 

-. :  erreur  de  xf 

Globalement,  l'erreur  quadratique  moyenne  est  de  erf  = 
3,413  pour  la  fusion  qui  ne  prend  pas  en  compte  le 
contexte  et  seulement  de  ere  =  0,806  pour  la  fusion  qui 
prend  en  compte  le  contexte.  On  constate  que  le  gain  est 
tres  net  en  faveur  de  l'approche  proposee. 


4.3  Re  marques 

Bien  evidemment  les  deux  algorithmes  utilises  dans  la 
simulation  n'exploitent  pas  les  memes  informations  et 
la  comparaison  des  performances  est  de  ce  fait  biaisee. 
Une  comparaison  avec  des  algorithmes  calculant 
automatiquement  les  variances  des  signaux  observes  et 
qui  les  integrent  dans  les  gains  de  Kalman  serait  plus 
judicieuse  et  permettrait  de  comparer  des  algorithmes 
dont  les  fonctions  sont  plus  proches  l'une  de  l'autre. 
Toutefois,  la  encore,  cette  comparaison  ne  peut  etre  que 
partielle  puisque  les  sources  d'informations  ne  sont 
toujours  pas  rigoureusement  les  memes.  Dans  un 
premier  cas  I'information  ne  provient  exclusivement  que 
des  signaux  issus  des  differents  capteurs,  dans  le 
deuxieme  cas  I'information  est  toujours  composee  des 
signaux  des  capteurs  mais  aussi  de  parametres  exterieurs 
ou  bien  de  resultats  d’heuristiques  visant  a  evaluer  la 
qualite  de  la  mesure. 

Comme  nous  l'avons  remarque  toute  la  difficulty  dans  la 
definition  d'un  algorithme  pour  un  systeme  multicapteur 
reside  dans  la  demarche  qui  permet  d'accorder  une 
importance  a  chaque  capteur  dans  le  processus  de  fusion. 


26-11 


Dans  l'algorithme  presente  cette  importance  est  accorde 
au  vu  de  deux  criteres.  Le  premier,  d'ordre  statistique, 
vise  a  evaluer  la  qualite  du  signal  en  terme  de  variance. 
Ce  critere  est  deja  naturellement  pris  en  compte  dans  le 
filtre  de  Kalman  par  le  biais  des  matrices  de  covariance 
du  bruit  d'observation.  Le  second  critere  est  l'adequation 
du  capteur  au  contexte  et  revaluation  de  son  bon 
fonctionnement.  II  peut  subsister  une  certaine  confusion 
entre  ces  deux  criteres  puisqu'une  non-adequation  du 
capteur  au  contexte  peut  entrainer  une  modification 
statistique  du  signal.  Cela  pourrait  laisser  a  penser  que 
les  seuls  parametres  statistiques  sont  suffisant  pour 
evaluer  l'importance  du  capteur  dans  le  processus  de 
fusion.  Mais  la  relation,  reliant  la  presence  d'une 
perturbation  a  une  modification  des  parametres 
statistiques  du  bruit  n'est  pas  une  relation  biunivoque,  et 
un  capteur  peut  delivrer  des  mesures  d'une  qualite 
statistiquement  satisfaisante  mais  totalement  aberrante 
au  vu  des  objectifs  suivis. 

L'apport  de  la  methode  proposee  repose  done  bien  sur 
1’ identification  et  I'utilisation  de  toutes  les  informations 
qui  peuvent  etre  disponibles  au  niveau  du  systeme  pour 
evaluer  l'adequation  du  capteur  au  contexte,  sachant  que 
les  caracteristiques  du  bruit  sont  deja  naturellement 
prises  en  compte  dans  le  filtre  de  Kalman. 

5.  Conclusion 

Nous  avons  propose  un  algorithme  de  fusion  de  donnees 
permettant  de  tenir  compte  du  contexte.  Cette  prise  en 
compte  est  essentielle  pour  un  systeme  multicapteur  car 
elle  permet  de  ne  selectionner  a  tout  instant  que  les 
mesures  pertinentes  et  de  reduire  l'importance  ou 
simplement  d'exclure  les  mesures  qui  pourraient 
perturber  le  signal  utile.  Le  contexte  est  done  analyse  par 
un  niveau  de  traitement  que  Ton  pourrait  qualifier  de 
symbolique  puisqu'il  resulte  d'une  analyse  prealable,  par 
un  expert,  des  differentes  situations  pouvant  survenir 
dans  I'utilisation  du  systeme  et  des  moyens  a  mettre  en 
ceuvre  face  a  celles-ci.  Ce  niveau  de  traitement  permet  de 
superviser  des  traitements  classiques  de  traitement  du 
signal  qualifie  de  numerique.  Cette  approche  permet  de 
montrer  la  dualite  qui  existe  entre  les  deux  traitements  et 
illustre  la  synergie  qui  peut  en  resulter.  Outre  la 
poursuite  de  cibles  aeriennes,  nous  pensons  que  de 
nombreuses  applications  peuvent  beneficier  du 
formalisme  developpe  dans  cet  article. 

6.  Annexe 

Dans  le  cas  general,  la  demonstration  de  la  formule  (2), 
issue  de  la  formule  de  Poincare,  se  fait  par  induction  sur 

la  variable  k=IJ  1  et  sur  la  variable  n  =  INI  avec  k<n.  Le 
principe  est  de  demontrer  que  si  la  formule  est  verifiee  a 
l'ordre  (n-l,k-l)  et  a  l'ordre  (n,k-l)  alors  elle  est  verifiee 
a  l'ordre  (n,k).  Sachant  que  la  formule  est  verifiee 
quelque  soit  n  pour  tous  les  couples  (n,0),  l'induction  est 
complete  pour  tout  couple  (n,k)  tel  que  k<n. 


p(cj)  -  P(nieJ  Q  njEj  Cj ) 

A  l’evidence,  la  formule  est  verifiee  pour  tout  couple 
(n,0)  en  effet  celle-ci  s'ecrie. 

P(cn)  =  Cj  ) 

=  X  (-1)  II‘NIP(niEl  Cj) 

NCI 


Supposons  que  la  relation  soit  vraie  pour  le  couple 
d’indice  (n-l,k-l)  et  pour  le  couple  d'indice  (n,k-l).  Nous 
allons  montrer  qu'elle  est  vraie  pour  le  couple  d'indice 
(n,k). 

Soit  J  tel  que  IJI  =  n-k  et  IJI  =  k,  alors  : 

P(cJ)=P(nieJcinc1njEj_{1}Cj)  (Ai) 

ou  1  est  un  indice  tel  que  1  E  J . 

En  utilisant  la  relation  : 

P(AD  B  )  =  P(A)  -  P(AH  B) 

ou  A  et  B  sont  deux  sous-ensembles  quelconques,  (Al) 
peut  s'ecrire  : 

P(cj)=P(niejCinjej.(1}Cj)-P(nieJu{1}cinjEj_(1jCj ). 

Cette  relation  pouvant  aussi  prendre  la  forme  : 

P(cj)=X  (-i)iIJIP(ni£lci)-X  (-i)II'JI+1P(nieI  Q) 

JCI  JU{1)C  I 

If  I 

P(CJ)=X  (-i)II'JIP(niElci)+X  (-i)II'JIP(ni£l  Cj) 

JCI  JCI 

1$  I  le  I 

=  X  (-1)  IMIP(ni£l  CO, 

JC  I 

l'induction  est  ici  complete. 
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PAPER  No.  26 

DISCUSSOR’  S  NAME:  Y.  Nimier 

COMMENT/QUESTION : 

Les  bruits  de  mesure  de  differents  capteurs  sont  ils  reajustes  dynamiquement  en  fonction  soit  de 
l’environnement,  soit  des  modes  d’emission  mis  en  oeuvre  par  les  capteurs  (formes  d’ondes 
specifiques). 

(Is  the  measurement  noise  from  different  sensors  readjusted  dynamically  either  with  respect  to 
the  environment,  or  to  the  transmission  modes  used  by  the  sensors  (specific  waveforms). 


AUTHOR/PRESENTER’S  REPLY: 

Les  matrices  des  bruits  d’ observation  des  capteurs  peuvent  etre  modifies  automatiquement  en 
fonction  des  modes  d’emission.  L’environnement  est  pris  en  compte  par  le  haut  niveau. 

(The  observation  noise  matrices  of  the  sensors  can  be  modified  automatically  to  suit  the 
transmission  mode.  The  environment  is  allowed  for  by  the  high  level). 
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Abstract  :  un  systeme  de  fusion  de  donnees 
symboliques  a  pour  but  d’elaborer  et  d’entretenir  a 
tout  instant  une  image  la  plus  complete,  exacte  et 
coherente  possible  d'un  monde  exterieur  en  evolution, 
appelee  situation,  a  partir  d’informations  d’ entree 
incertaines,  incompletes  et  imprecises.  La  modelisation 
d'une  situation  necessite  la  representation  de  la  notion 
d'imprecision  sur  les  caracteristiques  des  objets  pergus, 
la  notion  de  doute  quand  a  l’existence  d’un  objet,  mais 
aussi  la  representation  de  l'information  hypothetique 
utilisee  au  cours  du  raisonnement.  Nous  presentons 
dans  cette  communication  une  approche  formelle 
permettant  d’integrer  de  maniere  unifiee  ces  differentes 
notions  dans  un  meme  modele  de  representation,  en  les 
dotant  d'une  semantique  precise.  Nous  decrivons 
ensuite  l'application  faite  de  cette  approche  et 
l'implementation  pilote  realisee  dans  le  cadre  du 
programme  international  DFD  (Data  Fusion 

Demonstrator). 

Mots  cles  :  Systemes  d'Information  et  de 

Communication,  Bases  de  Donnees,  Theorie  des 
Possibilites,  Fusion  de  donnees  symboliques, 

representation  des  connaissances,  approche  objet. 

L _ La  fusion  de  donnees  symboliques. 

La  fusion  de  donnees  est  de  maniere  generale  le 
processus  visant  a  elaborer  et  a  entretenir  1"' image"  la 
plus  fidele,  complete  et  precise  possible  d'un  univers 
exterieur  evolutif  a  partir  d'un  flot  d' informations 
partielles,  incertaines  et  bruitees.  Nous  considerons  ici 
la  fusion  de  donnees  dans  le  domaine  du  champ  de 
bataille  (determination  et  evaluation  des  forces 
ennemies  en  presence),  les  problematiques  soulevees 
pouvant  s'appliquer  plus  generalement  a  une  large 
classe  de  contextes  tels  que  securite  civile,  gestion  de 
crise,  maitrise  des  risques  industriels,  etc. 

Les  multiples  informations  en  entree  peuvent  etre  de 
nature  symbolique  (observations  humaines, 
informations  deja  partiellement  fusionnees  issues 
d'autres  systemes),  ou  numerique  (mesure  effectuee  par 
un  dispositif  physique  capteur,  tels  que  radar,  imagerie 
visible,  ecoute  radioelectrique,  etc.). 


De  maniere  generale,  les  sources  d'information  sont 
heterogenes  (elles  refletent  differents  phenomenes 
physiques  partiellement  independants)  et  asynchrones 
(elles  delivrent  des  informations  a  des  instants  non 
necessairement  predetermines).  Elles  ont  par  ailleurs 
des  champs  de  vision  limites  (elles  ne  pergoivent 
qu'une  certaine  partie  du  monde  exterieur  a  un  instant 
donne),  et  delivrent  des  informations  incertaines  et 
imprecises  (ce  sont  des  dispositifs  physiques  imparfaits 
par  nature).  L'image  generee  (en  constante  evolution) 
du  monde  exterieur  sera  dite  situation  per  que. 

Dans  le  fusion  de  donnees  dite  symbolique  a  laquelle 
nous  nous  interessons  ici,  on  suppose  que  tous  les 
signaux  en  entree  ont  ete  prealablement  transformes 
par  le  biais  de  traitements  convenables  en  donnees 
symboliques. 

Un  processus  de  fusion  peut  etre  mene  de  maniere 
manuelle,  semi-automatique  (avec  l'assistance  de 
traitements  informatiques),  voire  completement 
automatique1.  Ce  dernier  cas  fait  l'objet  du 
programme  international  de  recherche  DFD  (Data 
Fusion  Demonstrator)  dans  lequel  se  sont  inscrits  les 
travaux  ici  presentes.  Dans  ce  cadre,  un  probleme 
essentiel  a  alors  trait  a  la  modelisation  de  la  situation 
perque,  et  de  sa  dynamique  au  cours  du  temps.  Ce 
probleme  fait  l'objet  de  la  presente  communication. 

Nous  precisons  tout  d'abord  les  besoins  de 
representation  de  donnees  incertaines,  imprecises  et 
hypothetiques  dans  le  cadre  de  la  fusion  de  donnees 
symboliques.  Nous  presentons  ensuite  une  approche 
possibiliste  proposee  pour  la  representation  de  la 


1  Les  traitements  de  fusion  s'appuient  sur  un  ensemble  de  connaissances  a 
priori  refletant  les  invariants  et  la  dynamique  des  entites  du  domaine 
considere.  Dans  le  domaine  ici  considere,  on  trouvera  la  cartographie 
numdrique,  les  performances  d’equipcments  ennemis,  les  ordres  de  batailles, 
les  doctrines  ennemies,  etc. 

Mentionnons  enfin  que  le  processus  de  fusion  peut  etre  distribue  sur  plusieurs 
sous-systdmes  inddpendants  (travaillant  en  cascade  et/ou  en  parallele), 
appeles  ici  noeuds  de  fusion,  au  travers  desquels  transite  une  information  de 
plus  en  plus  en  plus  enrichie.  On  considerera  ici  non  pas  le  processus  complet 
mais  cc  qui  se  passe  au  sein  d'un  noeud. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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connaissance  imparfaite,  les  notions  de  mesures  et  de 
quantites  fonctionnelles ,  proposee  pour  lever  les 
limitations  de  celle-ci,  et  enfin  le  modele  de  donnees 
SAGESSE  construit  en  reponse  a  notre 
problematique  .  Enfin,  nous  decrivons  l'application 
faite  du  modele  dans  un  noeud  de  fusion  et 
l'implementation  realisee. 

2. _ Problematique. 

Dans  le  domaine  du  champ  de  bataille,  une  situation 
pergue  sera  la  description  d'un  ensemble  d'entites 
evoluant  au  cours  du  temps.  Dans  ce  cadre,  il  est 
necessaire  de  disposer  d'un  modele  permettant  la 
representation  de  collections  d'objets,  rendant  compte 
simultanement  de  differents  aspects  [Bou  92],  [Pol  93, 
96] : 

•  imprecision  sur  une  valeur  numerique.  Ex  : 

•  "V unite  X  est  a  20km  a  10%pres.  Son  effectif 

est  de  18  a  20  hommes. "  (cas  1 ) 

ou  symbolique.  Ex : 

•  "I'unite  Y  est  un  Bataillon  ou  bien  un 

Regiment  de  blindes. "  (cas  2) 

•  incertitude  sur  1' existence  d'une  entite.  Ex  : 

•  "L‘ observation  du  capteur  C  laisse  presumer 

l' existence  d'un  vehicule  en  (x,  y).  (cas  3) 

De  plus,  la  realite  operationnelle  necessite  d'operer  un 
certain  type  de  fusion  dite  au  fil  de  Veau,  Ou  tout 
nouvel  indice  est  integre  immediatement  et  "au  mieux" 
dans  la  situation  courante,  c'est  a  dire  dans  l'image  que 
se  fait  le  systeme  du  monde  exterieur  a  cet  instant2 3 
Dans  cette  hypothese,  que  nous  adopterons  dans  toute 
la  suite,  l'integration  d'un  indice  consiste  idealement  a 
rechercher  dans  la  situation  courante  l'entite  de  la 
situation  pergue  avec  laquelle  l'indice  entrant  doit  etre 
correle ,  puis  a  agreger  la  nouvelle  description  partielle 
d'une  entite  avec  un  etat  plus  ancien  et  deja  connu  de 
celle-ci. 

Le  caractere  a  la  fois  incertain  (une  information  peut 
se  reveler  fausse)  et  imprecis  (cas  par  exemple  de 
positions  et  vitesses  observees)  de  la  donnee  d'entree, 
jointe  aux  deplacements  inevitables  de  l'ennemi  entre 
deux  observations  entrament  de  nombreuses 
ambiguites  dans  l'integration  d'un  indice.  Ceci 
necessite  d'effectuer  des  hypotheses  successives  au 
cours  des  traitements,  qu'il  s'agit  alors  de  gerer  dans  la 
representation  de  la  situation.  Une  hypothese  peut 
bien  sur  dependre  d'une  hypothese  precedente  en 


2  Les  traitements  operes  sur  les  donnees  n’etant  pas  consideres  ici. 

3  Par  opposition  a  une  fusion  a  posteriori  ou  Ton  attendrait  d'avoir  collect^ 
l'ensemble  des  indices  afferents  a  une  certaine  tranche  de  temps  dans  une 
optique  d’optimisation  globale). 


liaison  avec  une  demarche  de  type  hypothetico- 
deductif. 

Dans  la  fusion  dite  "temps  critique"  ici  consideree,  les 
retours  arrieres  dans  le  raisonnement  sont  exclus,  ce 
qui  impose  de  representer  et  gerer  en  parallele 
plusieurs  description  alternatives  de  la  realiteJPol 
96].  Un  besoin  supplementaire  consiste  done  a  rendre 
compte  du  caractere  hypothetique  d'une  description 
(avec  expression  d'hypotheses  concurrentes  ou 
dependantes).  Ex  : 

"Il  est  possible  de  correler  l  'observation  <  <  unite 
vue  en  (x,y)>  >  avec  I'objet  de  situation  o1  ou 
encore  avec  I'objet  o?. 

•  dans  le  premier  cas ,  /  'etat  V1  de  Oj  devient 
V i  (et  I'etat  V2  de  02  reste  inchange). 

•  dans  le  second  cas,  c  'est  V2  qui  est  change 

en  V 2. "  (cas  4) 

La  pertinence  d'une  approche  possibiliste  plutot  que 
probabiliste  pour  representer  de  telles  descriptions 
imparfaites  est  aujourd'hui  largement  admise  [Bou  92], 
Dans  ce  cadre,  la  modelisation  de  l'incertitude  et  de 
l'imprecision  a  ete  largement  etudiee  a  travers  la  notion 
de  sous-ensemble  flou  [Zad  65],  et  la  theorie  des 
possibilites  [Zad  78],  [Dub  88],  On  trouve  de 
nombreuses  approches  pour  la  mise  en  oeuvre  de  ces 
concepts  au  sein  de  bases  de  donnees  relationnelles  [Tes 
84],  [Pra  84,  86],  [Pet  96]  et  dans  une  representation 
objet  [Zic  90],  [VGy  93],  [Sad  94].  Nous  avons  pour 
notre  part  deja  propose  de  tels  modeles  dans  le  cadre 
particulier  de  la  fusion  de  donnees  symboliques  [Pol 
93a,  94a,  b,  95], 

Nous  nous  proposons  ici  d'etendre  les  modeles  deja 
envisages  en  introduisant  tout  d'abord  une 
modelisation  possibiliste  de  situation  rendant  compte 
de  l'existence  incertaine  d'une  objet.  Nous 
presenterons  ensuite  une  seconde  extension  permettant 
la  representation  d'hypotheses  concurrentes  au  sein 
d'une  approche  formelle,  basee  sur  la  notion  de  mesure 
fonctionnelle,  ce  qui  permettra  de  manipuler  (creer, 
reponderer,  detruire)  les  descriptions  hypothetiques. 

3; _ Un  modele  de  representation  possibiliste  de 

situation. 

3,1.  Les  extensions  "nettes". 

On  considere  ici  les  situations  dont  la  structure  "ideale" 
(celle  du  monde  reel  que  Ton  veut  approcher)  peut  etre 
representee  par  une  base  de  donnees  modelisee  par  un 
ensemble  : 

B  =  {  (Si,  Ej)  ;  i  =  1,  ...n} 
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ou  S;  et  E;  definissent  respectivement 

I  'intention45  (structure  attributaire)  et  V  extension 
(ensemble  des  instances)  d'une  classe  Q.  Les  types  de 
structures  ici  consideres  sont  les  agregations  (produits 
cartesiens)  de  types  elementaires  pris  parmi  entier,  reel, 
types  enumeres,  reference  a  un  autre  objet4 5 6. 

Ex  :  Unite  =  (type_unite  :  enumeration  (Div, 
Reg,  Esc),  x :  reel,  y :  reel,  Vitesse :  reel) 

Une  extension  E;  sera  une  collection  d'instances  {  <  Oj, 
Vj  >  },  ou  : 

•  Oj  est  un  identifiant  d'objet  ( oid)  issu  d'un 
ensemble  O; 

•  Vj  est  un  vecteur  de  valeurs  du  domaine 
defini  par  E,(e space  V). 

Ex  :  E  =  {<  "1 ",  (Regiment,  50km  60km, 
40km/h>,  <  "2",  (Division,  20km,30km, 

20kmAi  >  } 

La  seule  contrainte  d'integrite  consideree  ici  est  celle 
d 'identite  d'objet,  soit : 

Vj  VV 2  =>  Oj  ^  02 

Un  telle  extension  ideale  est  done  ici  une  application  de 
O  dans  un  ensemble  V7.  L' ensemble  considere  est  ici  V 
exp  O.(notation  pour  V°).  Une  telle  extension  sera 
appelee  extension  nette. 

II  s'agit  maintenant  d'introduire  l'imperfection  de  la 
connaissance  dans  une  telle  description.  Celle-ci  peut 
porter  sur  les  valeurs  associees  a  un  objet,  ou  encore  se 
traduire  par  un  doute  sur  l'existence  de  celui-ci,  ou 
plus  generalement  sur  l'existence  d'un  sous-ensemble 
de  la  description. 

Nous  rappelons  dans  la  suite  comment  peuvent  etre 
introduits  dans  un  tel  modele  l'imprecision  (■ valeurs 
floues )  d'une  part  et  le  doute  existentiel  ( extensions 
floues)  d'autre  part.  Nous  presentons  ensuite  la 
generalisation  de  ce  modele  a  la  representation 
combinee  de  l'imprecision  et  du  doute  existentiel 
( extensions  floues). 

Ce  modele  sera  ensuite  etendu  a  la  representation 
hypothetique  ( extensions  hypothetiques) . 


40n  peut  construire  7  en  considerant  l'ensemble  [-1,1]”  muni  de  la  distribution 
de  possibilites  produit  resultant  des  distributions  partielles  p^tOj)  =  1  - 
I  oj  (dimention  finie),  ou  encore  l'espace  des  suites  infinies  a  valeur  dans  [-1, 
1  ]  (dimention  denombrable). 

5  Nous  emploieront  aussi  dans  la  suite  le  terme  de  domaine  de  valeurs,  plus 
explicite  que  celui  A'intention. 

6  Cette  vision  ignore  pour  l'instant  les  notions  propres  aux  bases  de  donnees 
telles  que  contraintes  d'integrite,  definition  de  methodes  d'accfes,  de  triggers, 
etc.  qui  sont  ici  hors  de  notre  propos. 

7  convenablement  etendu  avec  un  extra-element  representant  l'absence  d'objet 
pour  un  oid  donne. 


3.2.  Les  valeurs  floues. 

La  notion  d' imprecision  dont  est  entachee  la 
perception  de  la  realite  peut  etre  introduite  au  niveau 
des  valeurs  avec  la  notion  d 'ensemble  flou  sur  les 
domaines  de  celles-ci. 

On  considere  alors  des  extensions  de  la  forme  {<Oj, 
Vj  >},  ou  Vj  note  un  ensemble  flou  sur  V  [Pol  93], 
L'espace  ici  considere  est  alors  classiquement  ([0,  1]) 
exp  V)  exp  O. 

On  pourra  ainsi  representer  des  etat  d'objets  dont  un 
exemple  est : 

<o,  (type _unite= (Div,  Reg},  x—[49,  51],  y=[59, 
61],  vitesse  =  T(1 8,  19,  21,  22)  > 
ou  T(18,  19,  21,  22)  note  la  distribution  de  possibilites 
trapezo'idale  de  support  [18,  22]  et  de  noyau  [19,  21]). 

3.3.  L'existence  incertaine. 

Nos  besoins  imposent  de  pouvoir  representer 
l'incertitude  au  niveau  de  l'existence  meme  d'un  objet 
ou  d'un  ensemble  d'objets  dans  une  extension.  Pour 
representer  un  tel  doute  existentiel,  il  faut  maintenant 
considerer  des  ensembles  flous  dont  les  elements  sont 
non  plus  des  valeurs,  mais  des  collections  d'objets  (soit 
des  elements  de  [0,l]exp({0,l}expO)).  On  peut  doter 
cet  ensemble  d'une  structure  algebrique  basee  sur 
deux  lois  de  composition  internes  "  +  "  (alternative)  et 
(union  generalisee),  et  sur  un  produit  externe 
permettant  la  construction  et  la  mise  a  jour  de  telles 
collections  [Pol  94b,  95a] 

On  pourra  par  exemple  representer  des  composantes 
de  situation  pergue  de  la  forme  : 

S  =  1/({oiMo2})  +  0.8/{oi} 
qui  exprime  1' alternative  entre  deux  situations  reelles 
possibles  :  l'une  ou  Oj  et  o2  existent  (possibility  1), 
l'autre  ou  seul  oi  est  present  dans  la  realite  (possibilite 
0.8). 

Une  expression  de  la  forme 

a/ {o}  +  b/0,  ou  (a<  1  et  b  =  0)  ou  (a=  1) 

qui  represente  un  objet  d' existence  incertaine,  sera  dite 

binome  canonique  (un  tel  binome  est  idempotent).  Une 

forme  particuliere  d' extension,  dite  totalement 

factorisee,  sera  un  produit  de  la  forme: 

S  =  Pi  (a;/{0i}  +  b;/0) 

ou  tous  les  o;  sont  distincts. 

Dans  une  telle  representation,  des  valeurs  V  peuvent 
bien  sur  etre  associees  aux  objets.  Toutefois,  cette 
approche  ne  permet  d'associer  qu'un  vecteur  de  valeur 
unique,  ou  un  vecteur  flou,  a  un  objet  donne.  Ceci 
n'est  pas  suffisant  dans  la  mesure  ou  la  description 
pretee  a  un  meme  objet  peut  dependre  de  l'hypothese 
consideree  (cas  4  ci-dessus). 

Nous  proposons  ci-dessous  une  modelisation  plus 
riche  integrant  la  representation  simultanee  de 
l'imprecision  sur  les  valeurs  et  du  doute  existentiel. 
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3.4.  Les  extensions  floues. 

Une  extension  floue  sera  un  ensemble  flou  sur 
l'ensemble  des  extensions  nettes,  soit  un  element  de 
1' ensemble  B*  =  [0,  1]  exp  (V  exp  O). 

A  l'espace  V  des  valeurs,  on  peut  associer  l'espace  V* 
=  [0,  1]  exp  V  des  ensembles  flous  sur  V.  V*  peut  etre 
muni  des  lois  de  composition  internes  +  et  *,  et  d'un 
operateur  externe  "/",  definis  par8 : 

•  V*  =  Vi*  +  V2*  :  mv*  (v)  = 

Max  (mVi(v),  mV2(v))  (union) 

•  V*  =  V,*  *  V22  :  mv*  (v)  = 

Min  (mVi(v),  mV2(v))  (intersection) 

•  V*  =  A  /  Vi* ,  A  e  [0,  1]  :  my*  (v)  = 

Min  (A,,  mVi*(v)) 

Ces  lois  permettent  de  munir  B*  de  lois  de 
composition  internes  +,  *  et  d'un  produit  externe  / , 
definis,  si  Bj*  e  B* ,  B2  e  B* ,  A  e  [0,  1] ,  B  e  B.  par  : 

•  B*  =  Bt*  +  B2*:  mB#(B)  =  Max  (mBi(B), 
mB2(B))  (operateur  alternative  generalise) 

•  B*  =  Bx*  *  B2*  :  mB*  (b)  =  Sup  b  s  D  (bi,  bj) 
^  Min  (mB1*  (bj) ,  mB2*  (bj)) 

ou  D  est  la  fonction  associant  a  tout  couple  (bj, 
b2)  e  B2  d  'extensions  nettes  l'ensemble 
d' extensions  nettes  (e{0,  1}  exp  B)  defini  par  D 
(bi,  b2)  =  {EN(bt  D  b2)}  ou  EN  (b)  represente 
toute  extension  nette  dont  le  graphe 
fonctionnel  est  inclus  dans  b.  En  d'autres 
termes,  un  element  apparait  dans  Bi*  *  B2!;~  s'il 
apparait  dans  B1“r  et  pas  dans  B2*,  ou  vice-versa. 
Sinon,  il  apparait  dans  Bp'  *  B2:;‘  avec 
l'ensemble  des  valeurs  portees  dans  Bi’1-  ou  dans 
Bp',  excluant  les  valeurs  communes  (ou 
exclusif). 

•  B*  =  A  /  Bp' :  mB*  (b)  =  Min  (A,,  mBi*  (b)) 

On  verifie  un  certain  nombre  de  proprietes 
generalisant  celles  mises  en  evidence  dans  les  approches 
precedentes : 

•  +  et  *  sont  commutatives,  associatives  et 
possedent  chacune  un  element  neutre 
(distribution  nulle  et  0); 

•  *  est  distributive  par  rapport  a  + ; 

•  on  peut  immerger  l'ensemble  des  extensions 
nettes  muni  de  la  loi  u  dans  (B!;',  *).  Ce 
resultat  permettra  en  particulier  de  definir 


8  Les  operateurs  +  et  •  ont  avant  tout  un  role  formel  et  ne  doivent  pas  etre  vus 
comme  des  operateurs  de  fusion  (par  contre  ces  demiers  pourront  s'exprimer 
comme  des  combinaisons  des  premiers  jugees  pertinentes  dans  un  contexte 
particulier). 


des  operateurs  generalisant  sur  les 
extensions  floues  les  operateurs  insert, 
update,  delete  et  select  des  bases  de  donnees. 

On  verifie  egalement  les  identites  remarquables  : 

•  'k\/<Oi,  Vj>  +  A2  /  <oj,  V2>  =  (Aj  +  A2)  / 
<Oi,  V1+V2> 

•  A/ <o1;  Vj>  *  A2/<oj,  V2>  =  (Ai.A2)  /  <Ob 
V1,V2> 

ou  Aj  +  A2  et  Aj.Aj  notent  respectivement 
Max(A1;  A2)  et  Min(Ab  A2) 

•  plus  generalement 

Si  A;/ <0,  V;>  =  <0,  E  Ai/Vi> 
et : 

(E;  Ai/bi)  *  (Ej  Pj/bj>  =  £;  Ej  (A;  .  Pj)  bi 

•  b2 

Si  une  valeur  floue  V!:'  n'est  pas  normalisee,  et  si  elle 
apparait  dans  un  objet  e  sous  la  forme  e  =  A/<0,  V*>, 
alors  on  peut  la  remplacer  (en  notant  1 1  V|  |  = 

Sup{mv*(v)})  par : 

•  e  =  A/<0,  V'»>,  si  1  <  MvM,  ou  V'*  = 
AAV*  est  defini  par  : 

•  myt(v)  =  mv*(v)  si  mv*(v)  <  1 

•  mv(v)  =  1  sinon 

•  e  =  (A.I  I  Vl  I  )/<0,  V”!>,  si  1  >  IIVII,  ce 
qui  ramene  au  cas  precedent. 

ce  qui  donne  dans  tous  les  cas  l'identite  : 

A/<0,  Vs^>  ■  (1.1 1  V|  I  )/<0,  A  AV5t> 
qui  permet  de  ne  considerer  que  des  valeurs 
normalisees. 

L'exemple  A.' extension  floue  ci-dessous  : 

S  —  l/(< Oj,  Vi > » < o2)  V2 >)  +  0.8/(<O],  V'i>) 
permet  d' exprimer  l'alternative  entre  deux  situations 
reelles  possibles  :  l'une  ou  Oi  a  l'etat  V1  et  o2  a  l'etat  V2 
(possibilite  1),  l'autre  ou  l'etat  Vj  de  Oi  a  ete  modifie  en 
V'i  et  ou  o2  n'existe  pas  (possibilite  0.8).  Nos  resultats 
ci-dessus  permettent  d'operer  des  mises  en  facteur  de  S 
telles  que  : 

S  =  <Oi,  Vi.V'!>  *  (l/(<Oi,  Vi > ^ < o2, 
V2>)  +  0.8/(<oi,V'i>) 

qui  mettent  en  evidence  la  partie  "certaine"  de  la 

connaissance.  Si  V'iC  V]  (V'i  "precise"  Vi),  S  se  reduit 
\ 

a 

S  =  <oi,  V'i>  »  (1/(<Oj,  Vi-V'i>»<o2, 
V2>)  +  0.8/0) 

Une  forme  particuliere  simple  d  'extension,  dite 
totalement  factorisee,  sera  un  produit : 

S  =  P;  (a;/  <  Oj,  V;  >  +  b/0) 
ou  tous  les  o;  sont  distincts. 

Un  binome  de  la  forme  a/<o,  V>  +  b/0,  ou  (a<  1 
et  b  =  0)  ou  (a=l)  decrit  un  objet  d' existence 
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incertaine.  Le  cas  particulier  de  a;  =  1  ,  b;  =  0,Vi  est 
celui  d'une  extension  nette. 

On  demontre  par  ailleurs  qu'une  bijection  canonique 
permet  d'operer  1' identification  : 

[0,  1]  exp  (V  exp  O)  =  ([0,  1]  exp  V)  exp  O)  = 
[0,  1]  exp  (V  x  O) 

qui  autorise  a  considerer  indifferemment  une  extension 
floue  comme  : 

•  un  ensemble  flou  sur  1' ensemble  des 
extensions  nettes\, 

•  une  fonction  associant  a  tout  identifiant  une 
valeur  floue  V*, 

•  un  ensemble  flou  de  couples  (identifiant, 
valeur  > . 

Notre  formalisme  nous  permet  generaliser  les 
operateurs  classiques  des  bases  de  donnees.  Ainsi,  les 
operateurs  classiques  d 'insertion,  mise  a  jour,  et 
destruction  sont-ils  des  cas  particuliers  de  l'affectation  S' 
=  S  *  <o,  V*>  : 

•  insertion  :  S'  =  S  *  <  o,  V*  > ,  ou  o  n'est  pas 
dans  B; 

•  mise  a  jour  (a  prendre  au  sens  d'une 
reduction  d'etats9)  :  S'  =  S  *  <o,  V*>,  ou 
o  est  deja  dans  S  avec  une  "valeur"  V"tV::'; 

•  destruction  :  S'  =  S  *  <  o,  V*  > ,  ou 

o  est  deja  dans  S  avec  la  "valeur"  V*. 

La  mise  a  jour  avec  changement  de  valeur  ( correction ) 
peut  etre  decrite  par  : 

S'  =  S*  <o,  V*>  *  <o,  V'*> 

=  S*  <o,  V*AV'*> 

La  selection  pour  sa  part  est  exprimee  par  l'operateur 
sy  generalisant  la  restriction  (V1*  =  (...,  v'*b  ..)  est  un 
vecteur  de  valeurs  floues  definissant  un  critere  de 
restriction),  defini  par  :  -  - 

•  Sy(P;  <  Oi;  V*;  >  )  =  P;Sy( <  O;,  V5';  >  ) 

•  Sy(Si<Oi,  V*;>)  =  S;Sy(<0;,  V*;>) 

•  Sy(<  o,  v*>)  =  (v*.v*)/<  O,  V*>  + 
(V*±V'*)/0 

ou  V*.V'*  =  Sup;  (v*;  min  v'*j), 
et  V’LLV'*  =  Inf;  (v*i  max  (1  -  v'^;)) 

4.  Un  modele  de  representation  d'obiets 
hvpothetiaues. 

4.1.  Les  limites  de  l'approche  possibiliste. 

La  modelisation  precedente  permet  de  representer  des 
descriptions  alternatives  d'une  situation  reelle 


9  Consecutive  a  un  gain  d'information  pour  mieux  decrire  l'etat  au  meme 
instant  du  monde  reel. 

10  La  nouvelle  valeur  est  independante  de  l'ancienne.  Par  exemple  pour 

decrire  l'etat  a  un  instant  ulterieur. 


imparfaitement  connue.  Dans  la  fusion  au  fil  de  l'eau, 
une  situation  pergue  est  construite  par  raffinement 
successifs  a  partir  des  indices  re  jus  qui  viennent 
conforter  ou  infirmer  telle  ou  telle  eventualite 
formulee.  H  est  done  necessaire  de  faire  evoluer  les 
confiances  accordees  aux  hypotheses.  De  meme,  il  faut 
rendre  compte  des  liens  de  dependance  eventuels 
existant  entre  celles-ci,  de  maniere  a  pouvoir  retirer  des 
elements  de  connaissance  inseres  a  tort.  Dans  ce  cadre, 
l'approche  proposee,  qui  ne  represente  pas  la  notion  de 
dependance/independance  entre  hypotheses  n'est  pas 
suffisante. 

En  effet,  considerons  par  exemple  une  premiere 
situation pergue  S  =  S  =  <  Oj,  V>i'1>  ou  Ton  tient  pour 
certaine  l'existence  d'une  entite  oj  a  l'etat  (flou)  V’^11. 
Supposons  que  l'arrivee  d'un  nouvel  indice  laisse 
presumer  l'existence  (incertaine)  d'une  unite  (necessite 
0.2)  qui  peut  etre  (apres  calcul  de  correlation)  l'unite 
deja  perdue  (possibilite  1),  ou  une  nouvelle  (possibilite 
0.5).  Un  calcul  simple  donne  : 

S'  =  1/ < 0l,  V\>  +  0.5/(<Oi,  V!>* 

<o2,  V2>)+ 0.2/<o1,V1>  (1) 

que  Ton  peut  ecrire  aussi  :  1/  <  Oj, 

l/V'i  +  OJ/V^  +  0.5/(<ob  V1>»<o2,  V2>) 

Si  aucune  nouvelle  observation  ne  vient  conforter  la 
nouvelle  "piste"  o2  a  Tissue  d'un  temps  determine, 
alors  le  processus  de  fusion  peut  decider  de  diminuer  la 
confiance  en  son  existence  (possibilite  0.8  au  lieu  de  1). 
L'etat  de  la  connaissance  souhaite  est  alors  : 

S"  =  0.5/(<ob  V1>»<o2,  V2>)  + 

0.8/<ob  V'i>  +  l/<oh  Vi>  (2) 

=  0.5/(<ob  V1>*<o2,  V2>)  + 

1/  <  ob  O.S/V'i  +  l/V^ 

Le  passage  de  la  forme  (1)  a  la  forme  (2)  requiert  la 
resolution  d'un  systeme  d'equations  en  min/max  qui 
n'admet  en  general  pas  de  solution  unique.  On  integre 
en  effet  dans  un  reel  unique  des  elements  de 
provenances  differentes  et  dont  la  memoire  n'est  pas 
conservee.  Il  n'est  done  pas  possible  d'effectuer  a  partir 
de  (2)  la  transformation  souhaitee. 

Nous  presentons  dans  la  suite  la  notion  de  mesure 
fonctionnelle  permettant  de  lever  ces  limitations,  puis 
les  extensions  hypotbetiques  construites  sur  cette  base. 

4.2.  Mesures _ fonctionnelles. _ quantites 

hvpothetiques. 

Nous  presentons  brievement  ci-dessous  la  notion  de 
quantite  hypothetique  [Pol  95b],  [Rob  96].  Nous 
utilisons  pour  cela  une  notion  de  mesure  fonctionnelle 
apparentee  a  celle  de  L-fuzzy  set  de  [Gog  67],  et 
s'inscrivant  dans  une  modelisation  possibiliste. 


11  L'etat  d'un  objet  contient  ici  en  particulier  sa  position  geographique. 
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Ou  appellera  mesure  fonctionnelle  F  sur  un  domaine  de 
valeurs  D  une  application  de  P(D),  non  plus  dans  [0, 
1],  mais  plus  generalement  dans  un  treillis  (T,  +  ,  *),  et 
verifiant  : 

F(0)  =  0  ;  F(D)  =  1 

V  Pie  P(D),  V  P2e  P(D),  F(PiUP2)  = 
F(P  i)  +  F(P2) 

On  verifie  dans  ce  cas  que  PicP2  =>  F(P1)<F(P2).  Une 
mesure  fonctionnelle  F  peut  etre  completement 
determinee  par  l'application  f  de  D  dans  T,  dite 
distribution  fonctionnelle,  definie  par  f(a)  =  F({a}) 
(moyennant  une  hypothese  de  continuite  pour  les 
domaines  de  cardinal  infinis,  non  detaillee  ici). 

On  considere  ici  le  cas  particulier  ou  T  est  un  sous- 
ensemble  donne  de  P(O)12,  stable  pour  l'union  et 
l'intersection.  Les  operateurs  +  ,  et  .  <  sont  alors 
respectivement  u  ,  n  et  I  (0  et  1  sont  respectivement  0 
et  Q).  T  ne  sera  alors  en  general  pas  totalement 
ordonne.(contrairement  a  [0,1]). 

Si  Q  est  un  singleton,  P(Q)  se  reduit  a  {0,0} 
(isomorphe  a  {0,1}  muni  des  operateurs  ou  et  et),  et  F 
est  alors  une  valeur  de  verite  classique.  Si  T  est 
l'ensemble  des  sous-intervalles  de  [0,  1]  de  la  forme  [0, 
1],  muni  des  lois  \j  et  n,  alors  F  est  une  mesure  de 
possibilites.  (u  et  n  sont  alors  le  max  et  le  min).  Nous 
supposerons  dans  la  suite  que  T  est  stable  pour  la 
complementation  (algebre  de  Boole).  F(A)  sera  appele 
support  de  verite  de  A  Les  elements  de  T,  qui 
generalisent  la  notion  de  valeur  entre  0  et  1,  seront 
appeles  pseudo-scalaires. 

Si  on  considere  par  ailleurs  une  mesure  de  possibilites  P 
sur  Q,  on  verifie  que  P'  =  P  °  F  est  une  mesure  de 
possibilites  sur  P.  Si  H  I B,  on  notera  dans  la  suite  P(H) 
=  i  H| .  Cette  quantite  sera  appelee  possibilite  de 
l'hypothese  H. 

Une  mesure  fonctionnelle  sur  D  permettra  de 
representer  un  doute  sur  la  verite  d'une  information 
tout  en  conservant  la  notion  de  source  dont  celle-ci 
est  issue,  refletant  done  a  tout  instant  un  etat 
potentiellement  affinable  de  la  connaissance  acquise. 
11  est  alors  possible  en  particulier  de  reponderer  les 
confiances  associees  aux  sources  de  donnees,  la  memoire 
du  calcul  d'une  quantite  etant  conservee  dans  la 
quantite  elle-meme. 

Deux  pseudo-scalaires  Hi  et  H2  seront  dits  independants 
ssi  I  Hj.  H2I  .=  I  Hj  min  I  H2| .  (on  ecrira  Ind( Hi,  H2)). 
Cette  notion  d'independance  rend  ici  compte  de  l'idee 
de  sources  de  donnees  issues  de  phenomenes 
physiques  non  interactifs.  Une  famille  de  p  pseudo- 


12  ensemble  des  parties  d'un  ensemble  donne  fi. 


scalaires  Hb  ...,  Hp  sera  dite  independante  ssi  tous  les  H, 
sont  independants  deux  a  deux.  Pour  T  donne,  le 
nombre  maximum  n  (s'il  existe)  d'elements 
independants  sera  dit  dimension  hypothetique  de 
T.(dim(T))  Une  famille  independante  de  T  de  cardinal  n 
sera  dite  base  hypothetique  de  T.  Le  cas  de  [0,  1]  et  d'une 
mesure  de  possibilite  definit  la  dimension  1. 

On  definit  par  ailleurs  les  notions  suivantes  : 

•  une  mesure  F  sera  dite  propre  ssi  AnB  =  0  => 
F(A)  nF(B)  =  0.  Elle  sera  dite  sera  to  tale  ssi 
^•a;  e  p  F(Ai)  =  Q  Une  mesure  a  la  fois 
propre  et  totale  peut  etre  definie  par  une 
application  inverse  f  :Q~>  {0,  1}  associant  a 
tout  to  e  Q  une  valeur  de  D. 

•  Q  sera  dit  dense  ssi  V  H  e  T  tel  que  |  Hi  = 
?ie[0,  1],  Vp  <  X,  3  FT  tel  que  |H'|  =  p, 
avec  Ind(H',  H")  V  H"eT  tel  que  Ind(H, 
FI").  Le  plus  grand  element  (au  sens  de  c) 
verifiant  cette  propriete  pour  FI  donne  sera 
note  A,/H.(on  montre  que  ce  plus  grand 
element  existe) 

•  Q  sera  dit  normalise  ssi  p  <dim(T),  il  existe 
une  famille  independante  {H;}  de  cardinal  p 
telle  que  I  H;|  =  1  V  i. 

On  supposera  dans  la  suite  que  ces  proprietes  sont 
verifiees.  Dans  ce  cas,  on  remarquera  qu'a  partir  d'un 
support  donne  FI  extrait  d'une  famille  independante,  il 

est  toujours  possible  de  former  un  nouveau  support 
de  verite  de  possibilite  arbitraire  1  (1/H)  (si  m=0)  ou 
de  necessite  arbitraire  m  (H  +  m/~]H)  (si  1=1). 

On  appellera  quantite  hypothetique  sur  un  domaine  D 
une  telle  mesure  fonctionnelle  sur  un  D.  L'ensemble  de 
telles  quantites  sera  note  Hn(D).  De  maniere  generale, 
si  D  est  muni  d'une  structure  algebrique,  on  retrouve 
dans  Hn(D).un  certain  nombre  des  proprietes 
algebriques  de  D  [Pol  95b].  Par  exemple,  si  on  note  IR 
le  corps  des  reels,  (Ha(D  IR),  +,  .)  est  un  anneau 
unitaire  dans  lequel  on  peut  immerger  (IR,  +,  .). 
Cette  propriete  est  remarquable  puisqu'elle  permet 
d’operer  des  calculs  sur  des  "nombres  flous"  en 
conservant  des  proprietes  des  operations  sur  les  reels 
(distributivite,  existence  d'inverses)  invalides  sur  les 
reels  flous  classiques. 

4.3.  Les  extensions  hypothetiques. 

Nous  generalisons  done  ici  l'approche  precedente  en 
introduisant  la  notion  d 'extension  hypothetique.  On 
constate  que  l'ensemble  de  la  construction  deja  operee, 
et  definissant  les  extension  floues,  peut  etre  menee  en 
remplagant  les  valeurs  de  [0,  1]  par  les  valeurs  d'une 
algebre  de  Boole  T  telle  que  definie  en  3.  Une  extension 
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hypothetique .  sera  alors  un  element  de  B*  =  T  exp  (V 
exp  O).  La  bijection  canonique  analogue  a  celle  deja 
enoncee  tient,  et  une  valeur  d'un  objet  pourra  done 
etre  consideree  comme  une  quantite  fonctionnelle  sur 
l'espace  des  valeurs  V.,  ce  qui  permettra  de  mener  des 
calculs  algebriques  sur  ceux-ci  respectant  les 
dependances  [Pol  93].  La  structure  d'algebre  de  Boole 
rend  compte  par  ailleurs  du  caractere  d'independance 
ou  de  dependance  entre  hypotheses,  ce  qui  leve  les 
limitations  mentionnees  ci-dessus. 

Un  binome  canonique  sera  de  la  forme  H/  <  o,  V  >  + 
lH/0  (1’  une  des  quantites  I  H I  et  |]H|  vaut  1),  et  une 
extension  totalement  factorisee  sera  de  la  forme  : 

S  =  Pi(H,/<o1,  V;>  +  lH;/0) 
ou  les  H;  constituent  une  famille  independante  au  sens 
donne  en  3.  Dans  la  pratique,  on  les  construira  a  partir 
d'une  base  hypothetique  {H,0}  unique  predefinie,  et 
Ton  utilisera  comme  deja  enonce  les  (1/H,°)  (si  m=0)  et 
(H;°  +  m/lH;0)  (si  1=1)  pour  former  un  couple  (H;, 
]Hj)  de  possibilites  arbitraires  (1,  m). 

La  presente  approche  n'est  autre  qu'un  calcul  formel 
sur  les  H;,  ~|H,  ,menant  a  des  calculs  de  possibilites  in 
fine,  les  constructions  mathematiques  proposees  en  3 
donnant  la  validite  necessaire  aux  operations  utilises  en 
leur  conferant  un  sens  rigoureux. 

5^ _ Application  a  un  noeud  de  fusion. 

On  considere  ici  un  noeud  de  fusion  en  montrant 
1' utilisation  qui  a  ete  faite  de  1' approche  precedente 
dans  le  cadre  d'une  fusion  au  fil  de  Veau  et  temps 
critique  qui  est  l'objet  du  programme  DFD. 

Un  tel  noeud  peut  etre  vu  comme  un  ensemble 
ordonne  d'operateurs  appliques  successivement  a  un 
indice  entrant13,  visant  en  final  a  integrer  celui-ci  dans 
la  situation  perdue  locale  du  noeud.  Ces  operateurs  sont 
[Pol  96]  : 

•  alignement  :  mise  dans  un  referentiel 
d'espace  et  de  temps  homogene; 

•  correlation  :  recherche  des  elements  de  la 
situation  perque  pouvant  correspondre  a 
l'entite  du  monde  reel  designee  par  l'indice 
entrant; 

•  association  :  choix  du  ou  des  elements  les 
plus  vraisemblables,  avec  creation  ou 
destruction  eventuelle  d' hypotheses; 

•  identification  :  enrichissement  maximum 
de  1'etat  en  utilisant  les  connaissances  a 
priori  sur  les  caracteristiques  des  objets 
observes  (la  connaissance  de  certains 
d'attributs  d'etat  permet  par  exemple  de 
raffiner  la  categorie  de  l'entite). 


13  Les  problemes  de  concurrence  d'accfes  a  la  situation  locale  sont  leves  en 
excluant  le  traitement  concurrent  d’indices  par  le  systeme  a  un  instant  donne. 


•  fusion  :  mise  a  jour  eventuelle  de  la 
situation  locale  en  fonction  de  seuils 
predetermines,  et  generation  en  sortie  du 
systeme  d'un  indice  refletant 
1' enrichissement  de  la  situation  opere  par  le 
noeud. 

Ces  operateurs  sont  completes  par  un  gestionnaire 
d'hypotheses  ayant  en  charge  d'une  part  la  mise  a  jour 
des  valeurs  de  confiance  accordees  aux  hypotheses  en 
fonction  des  delais  ecoules,  et  d' autre  part  la 
suppression  des  hypotheses  descendant  en  de$a  d'un 
seuil  de  confiance  determine. 

La  situation  locale  peut  etre  representee  par  une 
extension  hypothetique  d' objets  d'une  classe  nommee  ici 
SituationElement,  definie  [Pol  96]  par  les  vecteurs  de 
valeurs : 

V  =  (c,  tv,  X),  ou  : 

•  c  est  la  categorie  conceptuelle  d'appartenance 
(element  d'une  enumeration  ou  d'un  treillis 
de  symboles), 

•  tv  est  le  temps  valide,  e'est  a  dire  le  temps  du 

monde  reel  auquel  la  description  de  l'objet 
s' applique  (par  opposition  au  temps 

transactionnel  de  la  machine  auquel  cette 
description  est  consideree), 

•  X  est  l'etat  de  l'objet  proprement  dit 
comprenant  en  particulier  la  position  (x,  y), 

On  adjoint  d'autre  part  a  un  element  de  situation  la 
liste  des  indices  justifiant  1' existence  supposee  de  celui- 
ci.  Les  indices  sont  elements  d'une  extension  nette.  De 
plus,  un  objet  de  situation  fait  reference  aux  etat 
anciens  connus  de  celui-ci  pour  servir  de  base  a  des 
raisonnements  temporels. 

Une  situation  pergue  revetira  done  la  forme  : 

S  =  Expr(0,  <oh  Vj >,...,  <om  Vn>, 

ou  Expr  designe  une  expression  algebrique 
d'operateurs  +,  *  et  /. 

Une  situation  locale  est  dite  heterochrone,  celle-ci  se 
composant  d'etats  d'entites  ne  faisant  pas  reference  a  un 
meme  temps  unique  du  monde  reel.  La  correlation  d'un 
indice  avec  une  entite  de  la  situation  perque  fera  done 
appel  a  une  operation  Fc  dite  de  derivation,  permettant 
d'associer  a  un  etat  imprecis  V*  et  a  une  duree  donnee 
Dt  un  nouvel  etat  V'*  (en  general  plus  imprecis) 
contenant  l'ensemble  des  etats  possibles  atteignables  a 
partir  de  V*  dans  le  temps  Dt.  Une  telle  fonction  est 
specifique  de  la  categorie  conceptuelle  C  de  l'entite  et 
reflete  ses  capacites  dynamiques  (en  particulier  celles  de 
deplacement  dans  l'espace).  Une  telle  fonction  peut 
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egalement  faire  appel  a  la  description  du  milieu 
(terrain,  meteo)  si  celle-ci  existe. 

On  montre  que  chaque  etape  du  processus  de  fusion 
peut  etre  simplement  modelisee  a  l'aide  des  operateurs 
de  l'algebre  des  extensions  hypothetiques.  Ainsi,  par 
exemple,  si  un  indice  entrant  delivre  par  une  source  S 
est  represente  par  un  quadruplet  (E*,  tobs,  S,  1),  ou  E::' 
est  l'etat  observe  (imprecis),  tobs  le  temps  (monde  reel) 
de  1'  observation,  et  1  une  mesure  representative  de  la 
confiance  accordee,  celui-ci  sera  projete  en  un  element 
de  situation  : 

M  =  Hs/  <  o,  c*,  tv*,  X*  >  +  ]Hs/0 

•  Hs  est  une  nouvelle  hypothese  independante 
telle  que  P(HS)  =  1  ;  P(  |HS)  =  1-1 

(la  confiance  etant  ici  interpretee  comme 
une  necessite  d'existence  de  l'entite  decrite 
dans  la  situation14). 

•  c:'"  =  j(E*,  S)  ;  V*  =  y(E*,  S)  sont  donnee 
par  les  connaissance  a  priori. 

L’etape  de  correlation  necessite  pour  sa  part  l'operateur 
de  selection  appliquee  a  une  extension  deduite  de  la 
situation  pergue  par  application  de  l’operateur  de 
derivation. 

Toutes  ces  operations  son  "lineaires"  vis  a  vis  de  la 
somme  et  du  produit  externe  /,  sauf  1’ etape  de  fusion 
qui  est  non  lineaire  du  fait  de  I  utilisation  d'une  seuil. 

6. _ Implementation. 

Le  logiciel  SAGESSE  (Systeme  d'Acquisition  et  de 
Gestion  d'Elements  de  Situation  pour  les  SIC  Evolues) 
est  une  implementation  pilote  de  notre  modele  en 
C++  sur  station  UNIX.  Ce  logiciel  se  compose  du 
Systeme  de  Gestion  d'Objets  (SGO)  proprement  dit, 
auquel  nous  avons  adjoint  un  algorithme  de 
correlation  simple,  une  visualisation  sur  fond 
cartographique,  et  enfin  une  interface  usager  de 
manipulation  d'objets  permettant  l'expression  de 
requetes  sur  la  situation  perdue.  Un  generateur  de 
scenario  permet  d'alimenter  ce  noeud  de  fusion  en 
indices. 

Le  SGO  est  base  sur  une  extension  des  types  offerts  par 
le  modele  ODMG  [Cat  93]  par  des  types  tels  que 
fuzzy  _int,  fuzzy_float,  etc.,  H_int,  H_float,  etc. 
permettant  la  representation  d'attributs  dont  la  valeur 
est  imprecise  ou  peut  etre  conditionnee  par  une 
hypothese.  Les  definitions  de  classes  sont  alors 
exprimees  dans  le  langage  ODL  (Objet  Definition 
Langage)  de  l'ODMG,  convenablement  etendu. 

14  L'element  de  situation  Hj/<o,  V>  +  1h/0  avec  n(H)  =  1  ;  n(lH)  = 
ln’apportant  aucune  connaissance. 


Par  exemple  : 

interface  Unite  { 

H_extent  Unites ; 

H_string  nom; 

H_enum  {Escadron,  Regiment,  Division} 
categorie; 

H_int  effectif; 

H_location  position; 

Hrelationship  Set  <  Unite  >  subordonnes; 
H_relationship  Set  <  Indice  >  observations; 

} 

L'exemple  montre  un  nouveaux  types  d 'extension  de 
classe  representant  une  collection  fonctionnelle 
(implementees  comme  un  graphe  dont  les  feuilles  sont 
des  references  a  des  objets)  est  offert.  Une  extension 
specifique  des  collections  floues  existe  de  meme. 

Des  methodes  d'acces  et  de  comparaison  sont 
accessibles  depuis  l'interface  C+  +  d'une  part,  et  depuis 
le  langage  d'interrogation  base  sur  OQL  (Object 
Query  Langage)  de  l'ODMG  d'autre  part.  La  creation 
d'une  nouvelle  hypothese  (independante  de  celles  deja 
creees)  s'effectue  en  invoquant  le  "new"  (methode  de 
creation)  de  la  classe  "Hypothesis"  fournie. 

La  fonctionnalite  de  persistence  des  objets,  actuellement 
assuree  par  une  bibliotheque  logicielle  basee  sur  les 
Memory  Mapped  File  d'Unix,  (les  references  a  des  objets 
etant  classiquement  "traduits"  par  des  couples  (oid, 
pointeur  C)),  est  actuellement  en  cours  de  portage  sur 
le  SGBDOO  Objet  Store.  Les  nouveaux  litteraux  tels 
que  f  int,  etc.  sont  alors  simules  comme  de  nouvelles 
classes  , 

Les  recherches  en  base  necessitant  ordinairement  l'acces 
a  des  descriptions  alternatives  pouvant  etre  nombreuses 
(leur  nombre  croit  avec  le  nombre  d' hypotheses/sous 
hypotheses),  des  algorithmes  particuliers  utilisables 
lors  de  l'acces  a  l'information  sont  actuellement  etudies 
avec  une  estimation  de  leur  complexite.  Le  but 
recherche  est  de  conserver  les  temps  d'acces  du 
SGBDOO  lorsqu'aucune  hypothese  n’est  active,  ou 
encore  lorsque  la  situation  per9ue  est  un  produit  de 
facteurs  premiers  independants. 

7_. _ Conclusion 

Nous  avons  ici  defini  un  modele  de  donnees  adapte  aux 
futurs  systemes  de  fusion  de  donnees  symboliques, 
supportant  conjointement  dans  le  cadre  d'un  modele 
oriente  objet  les  concepts  d’imprecision  sur  les 
attributs,  de  doute  existentiel,  et  enfin  de 
representation  d'hypotheses  concurrentes.  Ce  modele 
est  dote  de  facilites  de  representation  des  donnees  et  de 

15  Systeme  de  Gestion  de  Bases  de  Donnees  Oriente  Objet. 

16  La  liste  des  types  de  litteraux  n'etant  pas  extensible  en  ODMG. 
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manipulation  de  celles-ci  au  cours  d'un  raisonnement. 
Notre  approche  permet  done  de  doter  un  systeme  de 
fusion  de  la  capacite  de  fonctionner  au  fil  de  l'eau  et 
en  temps  critique. 

D' autre  part,  une  approche  formelle  reposant  sur  une 
construction  mathematique  rigoureuse  dote  les 
differentes  facilites  proposees  d'une  semantique  precise. 
Ceci  nous  a  permis  d'etendre  les  operateurs  usuels 
d’insertion,  de  modification  et  d’ interrogation  d'une 
base  de  donnees. 

L'interet  de  ces  concepts  a  pu  etre  demontre  dans  le 
cadre  d'un  Systeme  de  Gestion  d'Objets  pilote, 
permettant  le  prototypage  d'un  noeud  de  fusion. 
Toutefois,  la  mise  en  oeuvre  du  logiciel  de  gestion 
propose  dans  le  demonstrates  du  programme  DFD 
necessite  encore  l'etude  d'algorithmes  d'acces  optimaux 
d'une  part,  et  le  portage  sur  un  veritable  SGBDOO 
d' autre  part,  ce  qui  est  le  sens  des  recherches 
actuellement  menees. 

Un  autre  point  d'interet  concerne  bien  sur  l'etude 
d' applicability  des  concepts  proposes  a  d'autres 
domaines,  en  particulier  issus  du  secteur  civil  (gestion 
de  crise,  maitrise  des  risques  industriels,  etc.). 
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PAPER  No.  27 

DISCUSSOR’S  NAME:  B.  Dacre-Wright 

COMMENT/QUESTION: 

Avez-vous  une  idee  de  la  robustesse  de  votre  methode  a  une  erreur  de  registration  entre  les 
images? 

(Do  you  have  an  idea  of  the  ruggedness  of  your  method  when  there  is  a  registration  error 
between  images?) 


AUTHOR/PRESENTER’S  REPLY: 

A  ce  jour,  nous  ne  nous  sommes  pas  interesses  a  la  robustesse  de  la  methode  a  une  erreur  de 
registration  entre  les  images.  Mais  c’est  quelque  chose  que  nous  comptons  analyser. 

(T o  date,  we  have  not  looked  at  this  aspect.  It  is,  however,  something  we  intend  to  analyse.) 


DISCUSSOR’S  NAME:  P.  Verlinde 

COMMENT/QUESTION: 

Est-ce  que  vous  avez  verifie  votre  hypothese  de  depart  sur  la  “normalite”  des  distributions  de 
probability? 

(Did  you  verify  your  initial  hypothesis  concerning  the  “ normality ”  of  the  probability 
distribution?) 


AUTHOR/PRESENTER’S  REPLY: 

Les  distributions  de  probability  ne  verifient  pas  l’hypothese  de  normalite.  Nous  avons  fait  une 
hypothese  simplificatrice  pour  utiliser  la  regie  de  decision  de  Bayes. 

(The  probability  distributions  do  not  check  out  the  normality  hypothesis.  We  made  a  simplified 
assumption  in  order  to  be  able  to  use  the  Bayes  decision  rule). 


ADAPTIVE  INTENSITY  MATCHING  FILTERS  :  A  NEW  TOOL  FOR  MULTI-RESOLUTION  DATA  FUSION. 
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1.  SUMMARY 

Many  different  multiresolution  fusion  methods  have 
been  proposed  in  the  literature.  An  important  actual 
aim  in  this  field  of  research  is  to  produce  colour 
composites  of  multiresolution  data  preserving  both 
the  essential  spatial  information  of  the  high 
resolution  image  and  the  spectral  information 
content  of  the  low  resolution  channels,  so  as  to 
produce  pseudo  high  resolution  spectral  channels 
which  can  be  further  processed  for  improved 
classification  or  other  information  extraction 
purposes. 

The  best  integration  results  in  this  regard  have 
been  obtained  by  the  HPF  algorithm  and  by  a  new 
fusion  method  based  on  multiresolution  analysis  of 
the  images  using  the  wavelet  transform. 

A  new  methodology  based  on  adaptive  intensity 
matching  filters  using  local  image  statistics  to 
spectrally  adjust  high  resolution  images  to  the 
radiometry  of  low  resolution  channels  is  described 
in  this  paper.  The  algorithm  tends  to  equalise  the 
mean  (LMM  algorithm)  or  the  mean  and  the 
variance  (LMVM  algorithm)  of  the  high  resolution 
image  with  those  of  the  low  resolution  channels,  on 
a  pixel  by  pixel  basis,  from  the  values  measured 
within  a  local  window  around  each  pixel  position. 

The  INR  (Intensity  Normalised  Ratio)  transform,  as 
defined  in  this  paper,  is  a  fast  alternative  to  the 
RGB-1  HS-RGB  transform,  and  allows  an  efficient 
implementation  of  the  intensity  matching  fusion 
method,  generalised  to  images  with  more  than 
three  channels. 

These  algorithms  are  applied  to  a  1024  x  1024 
window  extract  of  a  high  resolution  (5m)  KOSMOS 
KVR  1000  panchromatic  image  to  be  fused  with  a 
registered  low  resolution  (20  m)  SPOT  XS  image, 
obtained  over  the  city  of  Liege  (Belgium). 

The  results  obtained  for  varying  filtering  window 
sizes  are  compared  with  the  HPF  filter  and  the 
wavelet  transform  applied  to  the  same  set  of 
images. 


2.  INTRODUCTION 

The  terms  «fusion»,  «integration»,  «merging», 
«bandsharpening»  have  been  used 
interchangeably  in  the  scientific  remote  sensing 
literature  to  describe  processes  aiming  at 
condensing  information  coming  from  multisource 


and  multitemporal  satellite  image  data.  More 
specifically,  these  processes  aim  at  producing 
colour  composites  preserving  in  the  best  possible 
way  most  of  the  significant  or  desired  information 
contained  in  the  separate  images  or  image 
channels.  Clearly,  a  choice  as  to  which  information 
will  be  relevant  of  the  final  product  has  to  be  made. 

In  this  regard,  an  important  effort  was  realised  in 
recent  years  in  order  to  preserve  the  highest 
possible  spatial  resolution  of  the  available  data  sets 
without  destroying  spectral  information  content. 

The  most  usual  situation  in  this  field  is  the 
integration  of  three  spectral  channels  with  an  high 
spatial  resolution  image. 

The  onset  of  this  type  of  research  started  as  soon 
as  multisource  data  with  different  spatial 
resolutions  became  available.  The  best  known 
examples  concern  the  merging  of  SPOT 
Panchromatic  images  with  LANDSAT  TM  or  SPOT 
XS  channels.  These  first  methods  applied  simple 
arithmetic  operations  between  channels  or  used 
principal  component  analysis  (PCA)  to  condense 
information  ([1],  [2],  [3],  [4]).  The  final  products 
correspond  in  general  to  high  spatial  resolution 
images,  but  with  important  alteration  of  the  spectral 
information  content  of  the  original  channels  [2], 

A  first  step  toward  effective  spectral  information 
conservation  was  introduced  with  the  RGB-IHS- 
RGB  transform  ([5],  [6],  [7]).  This  transform  is 
performed  in  three  steps  :  (1)  transform  three  low 
resolution  multispectral  channels  from  RGB  to  IHS 
space  ;  (2)  replace  the  intensity  channel  with  the 
high  resolution  channel ;  and  (3)  perform  a 
backward  projection  from  IHS  to  RGB  space.  The 
method  preserves  two  of  the  essential 
characteristics  of  the  spectral  colour,  namely  the 
original  hue  and  saturation  of  the  three  spectral 
channels.  The  quality  of  the  final  product  however 
depends  on  the  actual  differences  between  the 
original  intensity  channel  and  the  intensity  of  the 
high  spatial  resolution  image.  Similar  hues  with 
identical  saturation  values  are  still  strongly  affected 
by  important  differences  in  intensity  values,  and  are 
therefore  not  always  similar  looking  on  the  final 
colour  composite.  The  method  performs  best  if 
there  is  an  evident  correlation  between  the  IHS 
intensity  component  and  the  high  resolution 
channel,  ensuring  that  bright  objects  on  the  original 
image  remain  bright  in  the  merged  product  [7], 
Unfortunately,  high  correlation  has  no  reason  to  be 
expected  in  most  situations,  where  the  images  to 
be  fused  stem  from  different  sensors  and  have 
been  obtained  at  different  times. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  "Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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To  overcome  this  lack  of  correlation  between  high 
and  low  resolution  channels,  a  simple  integration 
method  consists  in  adding  the  result  of  a  high 
pass  filter  (HPF)  of  the  high  resolution  channel  to 
the  spectral  channels,  as  was  already  proposed  by 
Showengerdt  [8],  This  high  pass  filter  is  directly 
related  to  the  structural  information  content  of  the 
high  resolution  image  and  removes  most  of  its 
spectral  information,  it  has  a  zero  mean  value  and 
can  be  integrated  in  the  low  resolution  spectral 
channels  by  calculating  appropriate  weighted 
sums,  so  as  to  ensure  minimum  modification  of  the 
low  resolution  spectral  channels  ([8],  [9]).  Chavez 
et  at.  [2]  showed  that  the  results  generated  with  the 
HPF  method  are  less  distorted  than  those  obtained 
by  the  traditional  IHS  or  PCA  methods. 

The  same  approach  -  injecting  high  spatial 
information  into  the  low  resolution  spectral 
channels  -  has  also  been  proposed  by  Ranchin  [10] 
and  others  ([11],  [12]),  and  has  been  developed  by 
means  of  multiresolution  analysis  of  remotely 
sensed  images  using  the  wavelet  transform  [13]. 
Undoubtedly  this  approach  produces  the  best 
results  in  regard  of  preservation  of  spectral 
information  content  of  the  low  resolution 
multispectra!  channels  [14]. 

This  paper  proposes  a  different  approach,  pursuing 
the  same  goals  and  achieving  similar  results  as  the 
multiresolution  wavelet  transform,  and  is  based  on 
adaptive  intensity  matching  techniques  using  local 
image  statistics. 

In  the  following  we  shall  briefly  describe  the  wavelet 
merging  method  principle  before  introducing  the 
adaptive  intensity  matching  approach.  These  two 
methods  and  the  HPF  method  will  next  be 
compared  as  applied  to  the  same  set  of 
multitemporal  and  multiresolution  data. 

3.  DATA  FUSION  BY  MULTIRESOLUTION 
ANALYSIS  AND  WAVELET  TRANSFORM. 

This  first  method,  called  the  ARSIS  method 
(Augmentation  de  la  Resolution  Spatiale  par 
Injection  des  Structures  )  as  described  by  Ranchin 
et  a!.  [14],  aims  at  injecting  high  spatial  structural 
information  from  the  high  resolution  image  into  the 
low  resolution  channel. 

The  multiresolution  analysis  by  means  of  the 
wavelet  transform  used  to  perform  this  task  is 
essentially  a  four  stage  process  (see  figure  1, 
adapted  from  Garguet-Duport  ©f  a/.  [11]:  (1 ) 
histogram  matching  of  the  high  resolution  image  to 
the  corresponding  low  resolution  channel  in  order 
to  adjust  radiometry  and  improve  the  initial 
correlation  between  the  two  images.  (2)  perform  a 
forward  wavelet  transform  of  the  high  resolution 
channel  so  as  to  produce  four  new  images  at  the 
same  resolution  as  the  low  resolution  image.  Three 
of  these  images  contain  the  structural  information 
of  the  high  resolution  image  to  be  injected  into  the 
low  resolution  channel.  They  are  coded  by  the 
wavelet  coefficients  and  correspond  to  directional 
high  pass  filters.  The  fourth  image,  a  low  pass 


filter,  corresponds  to  the  resampled  high  resolution 
image  at  the  resolution  of  the  low  resolution 
channel.  (3)  registering  the  low  resolution  image  to 
the  same  system  of  reference  as  the  high 
resolution  image.  And  (4),  apply  the  inverse 
wavelet  transform  to  the  registered  low  resolution 
channel,  to  produce  a  pseudo  high  resolution 
spectral  channel  with  the  added  structural 
information  content. 

In  order  to  produce  a  spatially  enhanced  colour 
composite,  this  process  is  successively  applied  to 
the  three  separate  spectral  channels  to  be 
combined. 


4.  DATA  FUSION  BY  ADAPTIVE  INTENSITY 
MATCHING  FILTERS 

As  mentioned  above,  the  quality  of  spectral  colour 
preservation  with  the  IHS  transform  depends  on  the 
difference  between  the  IHS  intensity  channel  and 
the  high  resolution  channel.  High  correlation 
between  these  two  channels  will  usually  produce 
good  results  [7],  If,  as  in  most  instances, 
correlation  is  poor,  then  the  only  remaining 
alternative  is  to  minimise  the  difference  between 
the  high  resolution  channel  and  the  IHS  intensity 
channel.  More  precisely,  the  original  spectral 
information  of  the  low  resolution  channels  will  be 
better  preserved  if  the  high  resolution  channel  is 
made  to  resemble  the  IHS  intensity  channel. 

Image  resemblance  can  be  achieved  by  matching 
the  histograms  of  the  two  images,  and  is  used  for 
instance,  to  minimise  radiometric  contrast  between 
neighbouring  images  in  a  mosaic  [15].  Another 
approach,  also  producing  similar  histograms,  is  to 
use  normalisation  functions  [16],  which  tend  to 
equalise  the  means  or  the  means  and  variances  of 
the  two  images.  Experience  shows  however  that 
these  approaches  are  strongly  case  dependant  and 
that  they  provide  the  best  results  when  the  initial 
image  correlation  is  high.  This  is  because 
histogram  matching  algorithms  only  remodel  the 
image  frequency  distribution  without  any  control  on 
the  actual  spatial  distribution  of  the  intensity  values 
within  the  images. 

In  order  to  ensure  true  intensity  matching  of  the 
images  spatially,  one  has  to  be  able  to  control  and 
remodel  the  frequency  distributions  at  a  local  scale. 
This  can  be  performed  by  using  adaptive  filtering 
techniques,  which  have  also  widely  been  used  in 
other  domains,  such  as  speckle  removal  from  radar 
images  ([17],  [18])  or  for  image  reconstruction 
purposes  of  lost  data  [19], 

Following  this  approach,  two  local  intensity 
matching  filters,  have  been  devised,  based  on  the 
normalisation  functions  of  Joly  [16],  one  adjusting 
the  local  means  of  the  images  (LMM)  and  one 
adjusting  both  local  means  and  variances  (LMVM). 
The  general  Local  Mean  Matching  and  Local  Mean 
Variance  Matching  algorithms  to  integrate  two 
images,  a  high  resolution  image  (H)  into  a  low 
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resolution  channel  (L)  resampled  to  the  same  size 
as  H,  are  given  by  : 

•  the  LMM  algorithm 


•  the  LMVM  algorithm 


(DN, 


H>L(l,c) 


H(l,c)  J^g(/,c)XCrZ,(/,c)) 


+  M 


W,c) 


(2) 


where  M  corresponds  to  the  local  means  of  the 
images : 


Mi 


Im  (l,c) 


c+w/2  l+h/2 

Z  Z  dn, 

k=c-w/2  j=l-h/2 

hw 


Im(A-j) 


(3) 


H>L(l,c) 


(DNHgc)  ^H(l,c))aUl,c) 

<TH(1,c) 


+  DNL(1,c 


(6) 

The  resampling  process  of  the  low  resolution  image 
to  the  same  scale  as  the  high  resolution  image 
drastically  reduces  its  local  standard  deviations 
(ctl(i,c)  «  0),  especially  if  the  resolution  ratio  between 
the  images  is  high  and  for  small  filtering  window 
sizes.  In  this  instance,  equation  (6)  simply  reduces 
to  the  limiting  case  : 


FH>L(1,c)  ~  DNL{l,c)  (7) 

The  fused  image  will  not  significantly  differ  from  the 
low  resolution  image.  In  this  case,  larger  local 
environments  should  be  taken  to  perform  the 
filtering  in  order  to  incorporate  more  information 
from  the  high  resolution  image. 

Without  the  constraint  of  variance  equalisation, 
equation  (6)  becomes : 


and  a  to  their  local  standard  deviations  : 


Fh>l(i,c)  *  ~  mhq,c)  +  DN L(l  c)  (8) 


c+w/2  l+h/2 

2  I  DNlikjy 

.k=c-w/2j=l-h/2 


c+w/2  l+h/2 

2  2  DN tm(£  ;) 

M=c~w/2j^l-h/2 


hw 


(4) 


measured  within  a  window  of  size  h*w  centred  on 
the  image  pixel  at  co-ordinates  (l,c),  DN 
corresponding  to  the  pixel  values  of  the  images. 

These  algorithms  will  produce  a  simulated  high 
spatial  resolution  image  (F)  pertaining  the  spectral 
characteristics  of  the  low  resolution  channel  (L). 
How  well  the  spectral  values  are  preserved  will 
depend  on  the  size  of  the  filtering  window.  Small 
window  sizes  produce  the  least  distortion.  Larger 
filtering  windows  incorporate  more  structural 
information  from  the  high  resolution  image,  but  with 
more  distortion  of  the  spectral  values. 

If  the  spatial  resolution  ratio  between  the  two 
images  is  important,  then  the  pixel  values  of  the 
resampled  low  resolution  image  will  already 
correspond  to  a  local  average  value  of  the  image. 
Hence,  in  this  case,  the  LMM  algorithm  (1)  can  be 
rewritten  as  : 


H>L(lc) 


DN  H(lc*DN L(l,c) 


M 


me) 


(5) 


which  shows  that  the  LMM  filtered  image 
corresponds  to  the  low  resolution  channel 
multiplied  by  the  image_to_local_mean  ratio  of  the 
high  resolution  image. 

The  LMVM  algorithm  becomes  : 


and  corresponds  to  the  HPF  integration  filter 
defined  by  Showengerdt  [8],  This  formulation  also 
corresponds  to  the  additive  form  of  equation  (5). 


4.1.  Procedural  steps  for  implementing  image 
fusion  by  adaptive  intensity  matching. 

As  described  earlier,  the  LMM  and  LMVM  filters 
perform  an  intensity  matching  of  two  channels 
which  have  previously  been  registered.  Hence,  this 
process  can  be  applied  between  the  high  resolution 
channel  and  three  individual  low  resolution  spectral 
channels  producing  three  spatially  enhanced 
images  which  can  then  be  combined  to  form  the 
desired  colour  composite. 

Instead  of  applying  the  filtering  algorithms  to  each 
of  the  separate  channels,  a  simplified  method  can 
be  adopted  based  on  the  IHS  conversion.  If  three 
channels  have  to  be  merged  with  a  high  spatial 
resolution  image,  these  channels  can  first  be 
converted  from  RGB  to  IHS  space.  The  high 
resolution  image  will  then  be  intensity  matched  to 
the  IHS  intensity  channel  and  substituted  to  this 
channel  before  a  reverse  projection  back  into  RGB 
space.  The  resulting  RGB  channels  correspond  to 
the  spatially  enhanced  spectral  channels,  and  are 
obtained  after  only  one  filtering  process. 

4.2.  Generalising  the  data  fusion 
implementation  with  the  INR  transform 

An  improvement  of  this  implementation  method, 
leading  to  a  further  simplification  and  a 
generalisation  of  the  procedure  to  more  than  three 
channels,  has  been  devised.  It  is  based  on  a  new 
analytic  formulation  of  IHS  space. 
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GEOMETRIC  RELATIONS  IN  RGB-IHS  SPACE 


RGB  -  IHS  SPACE  NORMALISED  RATIO  PLANE 

_  yco  HUE-SATURATION  PLANE 


B  =  XS1 

RGB  coordinates  IHS  coordinates  INR  coordinates 
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When  an  IHS-RGB  transform  is  performed,  only 
the  intensity  channel  is  taken  into  account  and 
replaced,  while  the  hue  and  the  saturation  channels 
are  left  unchanged.  It  is  however  necessary  to 
compute  both  these  channels  as  they  are  needed  in 
the  reverse  conversion  from  IHS  to  RGB  space. 

The  relationships  between  RGB  and  IHS  space  are 
presented  on  figure  2. 

The  RGB  axes  of  the  RGB  cube  correspond  to  an 
hypothetical  SPOT  multispectral  image  (XS3,  XS2 
and  XS1  channels).  Crosscutting  the  RGB  cube, 
the  IHS  space  is  schematically  represented  by  the 
RGB  triangle,  perpendicular  to  the  Intensity  (I)  axis, 
which  can  be  defined  as  the  sum  of  the  three  XS 
channels  ([5],  [20],  [21]) : 

I  =  XSl  +XS2+XS3  (9) 

The  triangle  itself  corresponds  to  the  two 
dimensional  Hue-Saturation  plane.  The  expressions 
of  the  Hue  and  Saturation  co-ordinates  are  cyclic, 
and  depend  on  the  position  of  the  individual  colours 
inside  the  three  subtriangles  making  up  the  RGB 
triangle  [5],  We  can  express  Hue  on  a  scale 
ranging  from  0  (=  blue)  to  3  (=  0  =  blue)  where  the 
particular  values  1  and  2  correspond  respectively  to 
green  and  red  hues.  Saturation  can  be  expressed 
on  a  scale  of  0  (=  no  saturation)  to  1  (=  total 
saturation).  For  each  subtriangle  we  thus  have  : 

H,  =  (XS2  -XS3)/  (I  -  3XS3) 

H2  =  1  +  (XS3  -XS1)/  (I  -3XS1)  (10) 

lh  =  2+  (XSl  -  XS2)/  (I  -  3XS2) 

S,  =  (I  -  3XS3)  /  (I  -  2XS3) 

52  =  (I  -  3XS1)  /  (1  -  2XS1)  (11) 

53  =  (I  -  3XS2)  /  (I  -  2XS2) 

where  the  subscripts  refer  to  the  particular 
subtriangles  illustrated  on  figure  2. 

Individual  colours  plotted  onto  the  RGB  plane  by 
projection  from  the  origin  (O)  of  the  axes  can  also 
be  described  in  terms  of  the  normalised  ratios  of 
the  three  colour  components  (barycentric  co¬ 
ordinates)  as  shown  on  figure  2.  Hence,  description 
of  IHS  space  can  advantageously  be  replaced  by 
what  we  shall  define  as  INR  space,  standing  for 
Intensity-Normalised  Ratio  space.  Three  co¬ 
ordinates  are  necessary  to  define  this  space,  the 
usual  intensity  co-ordinate  and  two  normalised 
ratios  : 

1  =  XSl  +  XS2  +  XS3 

XS1N  =XS1  //  (12) 

XS2n  =XS2  // 

The  third  normalised  ratio  is  redundant  as  it  can  be 
deduced  from  the  first  two: 

XS3n  =XS3  //  =7  -XS1n-XS2n  (13) 

Performing  a  backward  transformation  of  IHS  to 
RGB  space  with  preservation  of  hue  and  saturation, 


amounts  to  say  that  the  normalised  ratios  of  the 
three  channels  are  also  preserved.  Hence,  if  a 
panchromatic  SPOT  image  (PAN)  has  been 
substituted  to  the  IHS  intensity  channel,  then  the 
backward  transform  should  produce  three  new 
channels  for  which  the  following  relations  hold  : 

Inev  =  PAN  =  XSlnew  +  XS2new  +  XS3nm  (14) 

and 

XSlnew  /PAN  =  XS1N  =  XSl  // 

XS2new  /PAN  =  XS2n  =  XS2 /I  (15) 
XS3nm  /  PAN  =  XS3n  =XS3 /I 

Rewriting  these  equations  yields  : 

XSlnew  -XSl* PAN/ 1 
XS2new  =XS2 •  PAN/ 1  (16) 

XS3nev  =XS3 *  PAN  / 1 

showing  that  a  data  integration  procedure  by  INR 
transform  only  requires  computation  of  the  global 
intensity  channel  of  the  three  low  resolution 
multispectral  channels  even  without  having  to 
compute  any  normalised  ratio  beforehand.  These 
relations  also  clearly  stress  the  fact  that  the  original 
channel  ratios  are  preserved  by  the  new 
multispectral  channels.  The  new  channels  are  of 
course  identical  to  those  obtained  by  a  classic 
RGB-IHS-RGB  transform. 

Furthermore  the  data  fusion  by  INR  transform  is 
easily  generalised  for  multispectral  data  of  more 
than  three  channels,  such  as  for  instance 
LANDSAT  TM  multispectral  images.  For  n 
channels  (ch)  to  be  upgraded  with  a  high  resolution 
image  (H)  we  thus  have  : 

n 

1  =  Y*chi  (17) 

;= i 

new_chj  =  ch,*  H  //  (18) 

By  this  method,  only  one  intensity  matching  filtering 
of  the  high  resolution  image  to  the  global  intensity 
channel  is  needed  to  produce  all  the  integrated 
upgraded  spectral  channels. 

Based  on  this  approach,  the  general  merging 
method  can  be  described  in  four  steps  as  illustrated 
on  figure  3. 

The  first  step  is  to  register  the  low  resolution 
multispectral  channels  to  the  high  resolution  image 
(H),  so  as  to  obtain  resampled  multispectral 
channels  (r)  overlaying  exactly  the  high  resolution 
image.  This  increases  artificially  the  spatial 
resolution  of  the  channels,  as  their  pixel  size 
diminishes,  but  without  increasing  their  spatial 
information  content,  so  that  they  still  have  to  be 
considered  as  low  resolution  channels.  These 
channels  are  then  summed  up  (step  2)  to  produce 
a  low  resolution  global  intensity  channel  (L)  to 
which  the  high  resolution  image  (H)  will  be 
intensity-matched  (step  3),  resulting  in  a  high 
resolution  global  intensity  matched  image  (F).  The 
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ADAPTIVE  INTENSITY  MATCHING 

METHOD  PRINCIPLE 
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i  one  high  resolution  channel 


1 

" 

j 

1 

L 

J _ 2 

tow 

resolution 


channel  H 

(high  resolution) 


GEOMETRIC 

CORRECTION 


n  resampled 
multispectral  channels 


r  n-1 


r3 


<(l/p)x(c/p» 


(Ixc) 


Registration  and  resampling  of  the  low  resolution 
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Adaptive  intensity  matching  algorithm 

•  Local  Mean  Matching  (LMM)  or 

•  Local  Mean  Variance  Matching  (LMVM) 
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final  step  applies  the  INR  transform  to  the  low 
resolution  resampled  channels  (ri)  using  the  F  and 
L  images,  yielding  the  simulated  high  resolution 
multispectral  channels. 


5.  DATA  DESCRIPTION  AND  APPLICATION 

The  three  fusion  algorithms  (LMVM,  LMM  and 
HPF)  were  applied  with  different  window  sizes  to  a 
SPOT  XS  scene  to  be  integrated  with  a 
panchromatic  KOSMOS  KVR  1000  image,  and  the 
results  are  compared  with  those  obtained  by  the 
ARSIS  method  computed  by  Dr.  T.  Ranchin  at  the 
University  of  Sophia-Antipolis  (Nice,  France). 

The  two  images  were  taken  above  the  city  of  Liege 
(Belgium),  respectively  in  May  and  in  June  of  1992. 
The  KOSMOS  image,  with  a  nominal  resolution  of 
two  meters  was  registered  to  the  1:25000 
topographic  basemap  and  resampled  by  cubic 
convolution  to  5  meters,  in  order  to  reduce  the 
original  resolution  ratio  between  the  two  sets  of 
data  from  1 0  to  4.  The  three  SPOT  XS  channels 
were  then  registered  to  the  KOSMOS  image,  and 
also  resampled  by  cubic  convolution  to  5  meter 
pixels.  A  square  window  extract  of  1024  x  1024 
pixels  over  the  south-eastern  part  of  the  city  was 
selected  to  perform  the  integration  analysis  (figure 
4.). 

This  study  area  along  the  Meuse  river  is 
characterised  by  recent  industrial  development  and 
suburban  growth.  Figure  4  shows  the  river  harbour 
bordered  by  industrial  zonings  to  the  south  and  to 
the  east,  urban  and  suburban  areas  to  the  north¬ 
west  and  mixed  forest  and  agricultural  areas  to  the 
south-east. 


Table  2  presents  the  correlation  between  the 
different  channels.  The  correlation  between  the  two 
channels  in  the  visible  part  of  the  spectrum  (XS1, 
XS2)  is  elevated,  but  is  low  between  the  visible  and 
infrared  part  (XS3)  of  the  spectrum.  Although  the 
two  images  were  taken  at  only  two  months  interval, 
the  KOSMOS  image  does  not  really  correlate  well 
with  any  of  the  XS  channels.  The  quite  low  KVR- 
XSI  correlation  value  precludes  a  satisfactory 
fusion  between  these  images  by  the  traditional  IHS 
method,  as  it  will  significantly  distort  the  spectral 
values  of  the  original  low  resolution  channels  [7], 


Table  2  :  Correlation  between  the  different 
image  channels 
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The  three  LMVM,  LMM  and  HPF  merging 
algorithms  were  applied  to  this  set  of  images  with  6 
different  window  sizes,  ranging  from  3  x  3  to  49  x 
49  pixels  in  size.  In  order  to  assess  the  quality  of 
the  different  fusion  algorithms  applied  to  the 
images,  a  set  of  statistical  parameters  based  on 
the  criteria  proposed  by  Ranchin  ef  al.  [14]  were 
computed  (tables  3  to  5)  and  are  illustrated  on 
figure  5.,  together  with  the  same  data 
corresponding  to  the  original  images  and  those 
obtained  by  the  ARSIS  method.  Ideally,  the  mean 
and  the  standard  deviation  of  the  integrated  and  of 
the  original  channels  should  remain  as  close  as 
possible.  Similarly,  correlation  between  these 
channels  shoud  be  as  near  as  possible  to  one  [14]. 


Table  1  summarises  the  main  statistical 
characteristics  of  the  three  SPOT  XS  channels  and 
of  the  KOSMOS  image.  The  XSI  channel 
corresponds  to  the  intensity  channel  of  the  IHS 
transform.  The  entropy  value  is  a  measure  of  the 
average  amount  of  information  of  the  channel,  as 
described  by  different  authors  ([22], [23]) : 

255 

Entropy  =  * log2 (p, )  (i9) 

;=0 


These  tables  also  lists  the  entropies  of  the  different 
results,  allowing  to  analyse  the  added  information 
content  of  the  integrated  channel  as  compared  to 
the  original  channel.  The  last  parameter  listed  in 
these  tables  is  a  deviation  index  (djndex)  as 
defined  by  Costantini  et  al.  [24] : 


\  l  C 

d  index  =  —  X  X 

lc  /= i  7=1 


ij 


(20) 


where  p,  is  the  probability  of  level  i  in  the  image. 

As  can  be  seen  in  table  1,  the  KOSMOS  image  is 
characterised  by  much  higher  standard  deviation 
and  entropy  values  than  the  corresponding  SPOT 
XS  channels. 


Table  1 :  Image  statistics  and  spatial  resolution 
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which  measures  de  relative  deviation  of  the  fused 
image  (F)  of  that  of  the  original  low  resolution 
reference  channel  (L).  The  smaller  this  value,  the 
better  the  image  quality. 

5.1  The  LMVM  filter  (table  3  ;  figure  5) 

By  construction,  the  mean  and  standard  deviation 
of  the  fused  images  remain  constant  for  all  filtering 
window  sizes.  Entropy  remains  constant  for  small 
window  sizes  (3  to  7).  Correlation  values  also 
remain  practically  constant,  very  near  the 
maximum  possible  value.  At  these  local  scales  the 
differences  between  the  reference  image  and  the 
fused  images  are  so  small  that  they  do  not  bear 
any  real  significance.  This  is  due  to  the  fact  that,  at 
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these  scales,  the  local  variance  values  of  the 
resampled  reference  image  (L)  are  nearly  zero 
because  of  the  quite  high  resolution  ratio  between 
the  two  images,  so  that  equation  (6)  simply 
reduces  to  the  limiting  case  of  equation  (7). 

With  increasing  window  size  (from  15  to  49), 
entropy,  correlation  and  deviation  index  start 
changing  significantly.  Entropy  and  deviation  index 
gradually  increase  with  window  size,  whereas 
correlation  values  decrease.  Referring  to  the 
numbers,  the  closest  match  to  the  ARSIS  filtered 
image  is  the  Local  Mean  Variance  Matched  image 
with  a  window  size  of  15,  although  bigger  window 
sizes  also  produce  good  results  with  still  quite  high 
correlation  values  (0.8  for  window  size  49). 

Visually,  the  fused  images  obtained  with  large 
window  sizes  (from  15  to  49)  remain  very  close  to 
the  ARSIS  filtered  image. 

5.2.  The  LMM  filter  (table  4,  figure  5  ) 

Here  too,  by  construction,  the  mean  of  the  fused 
images  remains  practically  constant  for  all  filtering 
window  sizes.  Standard  deviation,  entropy  and 
deviation  index  increase 

steadily  with  filter  window  size,  while  correlation 
decreases  continuously.  Comparing  the  numbers, 
the  ARSIS  filtered  image  lies  somewhere  in 
between  the  LMM3  and  LMM5  filtered  images. 
Visually,  for  these  small  window  sizes,  the  LMM 
algorithm  produces  images  the  most  similar  to  the 
ARSIS  image. 

5.3.  The  HPF  filter  (table  5,  figure  5) 

The  HPF  filter,  introduced  by  Showngerdt  [9], 
simply  adds  a  high  pass  filter  of  the  high  resolution 
image  to  the  resampled  low  resolution  channel.  The 
high  pass  filter  has  a  mean  value  of  zero  and 
therefore  preserves  the  mean  of  the  filtered  images 
for  all  filtering  window  sizes.  Standard  deviation, 
entropy  and  deviation  index  increase  rapidly  with 
filtering  window  size,  while  correlation  diminishes 
very  fast.  Although  considered  one  of  the  best 
integration  methods  so  far  [7],  it  does  not  compare 
favourably  with  the  three  other  integration  methods 
described,  producing  images  with  more  distorted 
spectral  values. 

6.  CONCLUSIONS 

This  study  has  shown  that  data  integration  of  a 
high  spatial  resolution  image  into  low  spatial 
resolution  channels  of  a  multispectral  image  can 
conveniently  be  performed  by  mean  of  intensity 
matching  the  high  resolution  image  with  the  low 
resolution  channels.  This  intensity  matching  is 
performed  by  locally  adjusting  the  mean  (LMM)  or 
the  mean  and  variance  (LMVM)  of  the  high 
resolution  image  to  that  of  the  low  resolution 
channels. 

The  spectral  values  of  the  original  low  resolution 
channels  are  the  best  preserved  when  filtering  the 
images  with  the  smallest  possible  window  sizes.  In 
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Figure  5 

LMVM,  LMM  and  HPF  filtered  image  statistics  in 
function  of  filtering  window  size.  XSI  is  the 
reference  image.  W  is  the  image  filtered  by  the 
ARSIS  method. 

the  case  of  the  LMM  algorithm,  window  sizes  of  3 
to  5  produce  the  results  which  compare  best  with 
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the  image  produced  by  the  ARSIS  method.  In  the 
case  of  the  LMVM  algorithm,  the  smallest  filtering 
window  sizes  produce  no  significant  difference 
between  the  original  channel  and  the  fused  image. 
Only  with  bigger  window  sizes  does  the  information 
content  of  the  fused  image  start  to  increase 
significantly  while  still  preserving  the  spectral 
characteristics  of  the  original  channels. 
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PAPER  No.  28 

DISCUSSOR’S  NAME:  B.  Dacre- Wright 

COMMENT/QUESTION: 

Quel  traitement  d’ interpolation  avez-vous  applique,  lors  du  reechantillonnage,  pour  eviter  les 
distorsions  spatiales  dans  l’image  obtenue. 

(Whcit  interpolation  processing  did  you  apply  during  resampling  to  avoid  spatial  distortions  in 
the  image  obtained?) 


AUTHOR/PRESENTER’S  REPLY: 

Tous  les  exemples  traites  et  illustres  ont  ete  reechantillonnes  par  une  convolution  bicubique,  qui 
est  la  moins  mauvaise  methode  de  reechantillonnage  utilisable. 

(All  the  examples  presented  were  resampled  by  means  ofbi-cubic  convolution,  which  is  the  least 
imperfect  resampling  method  available). 
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PAPER  No.  28 

DISCUSSOR’S  NAME:  E.  Schweicher 

COMMENT/QUESTION: 

1 .  Pourriez-vous  definir  ce  que  vous  entendez  par  capteur  a  faible  resolution  spatiale  et  a 
haute  resolution  spectrale. 

2.  Par  rapport  aux  differents  types  de  capteurs  que  vous  avez  cites,  dans  quelle  classe  de 
capteurs  mettriez-vous  les  radars? 

(I.  Could  you  define  what  you  mean  by  low  resolution  sensor  with  high  spectral  resolution? 

2.  Of  the  different  types  of  sensors  you  have  mentioned,  which  category  of  sensor  would  you 

put  radars  in?) 


AUTHOR/PRESENTER’S  REPLY: 

1 .  Dans  le  domaine  de  la  teledetection  spatiale  on  designe  generalement  par  haute  resolution 
spectrale  les  canaux  des  images  XS  (SPOT)  TM  (Landsat)  LISS  (IRS),  par  opposition  aux 
images  panchromatiques  qui  ont  une  tres  large  bande  passante  et  qui  ont  de  ce  fait  en  general  une 
meilleure  resolution  spatiale. 

2.  En  theorie  la  methode  de  fusion  proposee  permet  l’application  a  tout  type  d’images.  Un 
des  premiers  buts  de  cette  methode  est  d’ essay er  de  preserver  le  mieux  possible  T  information 
spectrale  d’origine  afin  de  produire  des  images  qui  seront  plus  facilement  interpretables  afin  de 
produire  des  images  qui  seront  plus  facilement  interpretables  ou  qui  seront  plus  facile  a  classer. 
L’information  spectrale  d’une  image  radar  est  “tres  different”  de  celle  du  domaine  visible  or 
infra-rouge,  et  ne  donne  pas  lieu  a  une  signature  spectrale  du  meme  type,  mais  en  principe  on 
peut  egalement  preserver  le  mieux  possible  cette  information  spectrale  radar  dans  le  cadre  de 

T  application  souhaitee. 

(1.  In  the  field  of  space  remote  sensing  we  generally  reserve  the  term  high  spatial  resolution 

for  image  channels  XS(SPOT),  TM(LANDSAT)  and  LISS(IRS),  as  opposed  to  panchromatic 
images  which  have  a  very  broad  bandwidth  and  which  as  a  result  have  a  better  spatial 
resolution. 

2.  In  theory,  the  fusion  method  proposed  can  be  applied  to  all  types  of  images.  One  of  the 
main  aims  of  this  method  is  to  try  to  preserve  the  original  spectral  data  in  the  best  possible  way 
in  order  to  be  able  to  produce  images  which  are  easier  to  interpret  or  to  classify.  The  spectral 
data  obtained from  a  radar  image  is  “very  different”  from  that  in  the  visible  or  infrared  domain 
and  does  not  produce  a  spectral  signature  of  the  same  type,  but  in  principle  this  radar  spectral 
data  can  also  be  preserved  in  optimum  fashion  by  the  application  in  question.) 
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PAPER  No.  28 

DISCUSSOR’S  NAME:  C.  Remus 

COMMENT/QUESTION: 

Existe-t-il  une  difference  en  grain  de  traitement  entre  la  methode  proposee  et  la  methode  ARSIS 
basee  sur  des  calculs  d’ondelettes. 

(Is  there  a  difference  in  grain  of  processing  between  the  method  proposed  and  the  ARSIS  method 
based  on  wavelet  calculations?) 


AUTHOR/PRESENTER’S  REPLY: 

La  rapidite  de  la  methode  ARSIS  ne  nous  est  pas  connu.  La  methode  de  fusion  par  ajustement 
des  moyennes  et  variances  locales  est  basee  sur  des  parametres  statistiques  simples  a  calculer,  et 
la  vitesse  de  calcul  peut  etre  rendue  independante  de  la  taille  de  la  fenetre  de  filtrage.  On  n’a  pas 
fait  de  test  precis  sur  V optimisation,  mais  c’est  tres  rapide. 

(We  have  no  information  on  the  speeed  of  the  ARSIS  method.  The  method  of fusion  by 
adjustment  of  averages  and  local  variances  is  based  on  statistical  parameters  which  are  simple 
to  calculate,  and  the  computation  speed  can  be  made  independent  of  the  size  of  the  filtering 
window.  We  haven  Y  run  a  specific  test  on  optimisation,  but  it ’s  very  fast.) 
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1.  SUMMARY 

The  aim  of  this  paper  is  to  explore  the  problem  of 
identity  data  fusion  in  the  context  of  naval  warfare 
where  commanders  and  their  staff  require  access  to 
a  wide  range  of  information  to  carry  out  their  duties. 
With  the  increasing  use  of  the  remote  sources  and 
the  increasing  use  of  information  that  has  already 
been  fused,  there  is  a  requirement  to  define  an 
identity  data  fusion  function  that  can  handle  organic 
and  non-organic,  local  and  remote  types  of 
information  characterized  by  different  accuracy  and 
timeliness.  This  paper  is  a  step  towards  the 
definition  of  an  architecture  that  can  perform  the 
fusion  of  organic  and  non-organic  identity  data 
through  the  use  of  statistical  analysis  rooted  in  the 
Dempster-Shafer  theory  of  evidence.  The  goal  is  to 
offer  the  decision  maker  a  quantitative  analysis 
based  on  statistical  methodology  that  can  enhance 
his/her  decision  making  process  regarding  the 
identity  of  detected  objects. 

2.  INTRODUCTION 

Research  and  Development  (R&D)  activities  by  the 
Data  Fusion  and  Resource  Management  (DFRM) 
Group  at  the  Defence  Research  Establishment 
Valcartier  (DREV)  are  aimed  specifically  at 
investigating  system  integration  concepts  for  the 
automation  of  Command  and  Control  (C2) 
processes  dealing  with  data  and  information 
management  and  tactical  decision  making  and 
action  implementation  issues  at  the  shipboard  level. 
The  group  has  developed  an  approach  to  help 


counter  the  anticipated  threat  to  our  surface  ships  by 
increasing  the  Above  Water  Warfare  (AWW) 
defence  capability  of  Canadian  Patrol  Frigates 
(CPFs)  through  the  development  of  a  real-time 
advisory  Decision  Support  System  (DSS)  (Ref.  1) 
that  supports  the  tactical  decision  making  and  action 
execution  processes  in  a  ship's  Operations  Room. 

The  increasing  tempo  and  diversity  of  open-ocean 
and  littoral  scenarios,  and  the  volume  and  imperfect 
nature  of  data  to  be  processed  under  time-critical 
conditions,  threats  characterized  by  high  speeds, 
low  approach  altitudes  or  steep  dive  trajectories, 
maneuverability,  and  the  ability  to  deceive 
defensive  systems  using  countermeasures  pose 
significant  challenges  for  future  shipboard 
Command  and  Control  Systems  (CCSs)  and  the 
operators  who  must  use  these  systems  to  defend 
their  ship  and  fulfill  their  mission. 

The  littoral  environment  is  especially  stressful.  The 
presence  of  dense,  commercial  air  traffic  and 
merchant  shipping  challenges  the  operators  in 
distinguishing  between  hostile,  neutral,  and  friendly 
tracks.  The  proximity  to  hostile  shores  decrease  the 
available  battlespace  and  warning  time  and  enable 
the  use  of  land-based  Electronic  Counter-Measures 
(ECM),  increase  clutter  levels  and  offer  more 
vulnerability  against  land-based  missile  attacks. 
Furthermore,  weapons  developers  recognize  the 
vulnerabilities  of  the  defensive  systems  and  develop 
or  evolve  weapons  in  an  attempt  to  defeat  them.  As 
the  threat  becomes  more  sophisticated,  additional 
aids  and  decision  support  to  the  operators  will  be 
increasingly  required. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  "Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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The  proposed  DSS  (Ref.  2)  would:  continuously 
take  in  data  from  the  ship’s  sensors  and  other 
information  sources;  form  and  dynamically 
maintain  an  accurate  AWW  tactical  picture,  and 
thereby  provide  the  most  likely  interpretation  of  the 
tactical  situation;  formulate  strategies  and  plans  for 
responding  to  anticipated  or  actual  threats, 
including,  as  necessary,  options  to  defend  the  ship 
using  the  best  possible  combination  of  hardkill  and 
softkill  weapons  or  other  defensive  means;  and 
present  fused  information  and  decision  support 
analysis  results  with  the  opportunity  for  the 
Commanding  Officer  and  AWW  team  to  accept  or 
reject  recommended  actions  and  plans  in  a  timely 
manner,  and  coordinate  and  direct  the 
implementation  of  actions  and  plans  that  result  from 
interaction  with  the  human  decision  maker. 

The  DSS  would  be  an  embedded  component  of  the 
ship's  combat  system,  integrated  within  the  CCS 
system,  providing  real-time  implementations  of 
functions  for  Multi-Sensor  Data  Fusion  (MSDF), 
Situation  and  Threat  Assessment  (STA),  and 
Resource  Management  (RM).  In  view  of  the 
functional  integration  involved,  we  call  this 
subsystem  of  the  ship's  CCS  an  MSDF/STA/RM 
system. 

This  paper  addresses  specifically  the  identity  data 
fusion  aspects  of  the  MSDF/STA/RM  system  that 
may  encompass  the  MSDF  and  STA  domains  such 
as  defined  in  Refs.  (1-3).  It  discusses  the  differences 
between  the  problem  of  fusing  attributes  (often 
from  local  and  organic  sources)  and  identity 
declarations  (most  of  the  time  from  non-organic  and 
remote  sources).  The  paper  also  addresses  the 
refinement  of  the  identity  declarations  in  the 
situation  assessment  process  by  the  use  of  other  a 
priori  information  as  well  as  the  information 
inferred  by  the  behavior  and  cluster  analysis 
functions.  The  use  of  statistical  analysis  rooted  in 
the  Dempster-Shafer  theory  of  evidence  is 
proposed.  The  paper  is  organized  as  follows. 
Section  3  presents  two  generic  models:  one  for 
MSDF  and  the  other  for  STA.  Section  4  the  kinds  of 
potential  identity  information  that  are  going  to  be 
fused.  In  Section  5  we  discuss  the  various 
architectures  for  identity  fusion.  Section  6  describes 
the  identity  information  fusion  process  for  organic 
and  non-organic  information. 


3.  GENERIC  MODELS  FOR  MSDF  AND  STA 
FUNCTIONS 

Throughout  the  1980s,  the  three  U.S.  military 
services  pursued  the  development  of  tactical  and 
strategic  surveillance  systems  employing  data 
fusion  and  supported  extensive  research  in  the  areas 
of  target  tracking,  target  identification,  algorithm 
development  for  correlation  (association)  and 
classification,  and  the  application  of  intelligent 
systems  to  situation  assessment  (Ref.  4).  The  large 
amount  of  fusion-related  work  in  this  period  raised 
some  concern  over  possible  duplication  of  effort.  As 
a  result,  the  Joint  Directors  of  U.S.  Department  of 
Defense  (DoD)  Laboratories  (JDL)  convened  a  Data 
Fusion  Subpanel  to  (1)  survey  the  activities  across 
all  services,  (2)  establish  a  forum  for  the  exchange 
of  research  and  technology,  and  (3)  develop  models, 
terminology  and  a  taxonomy  of  the  areas  of 
research,  development  and  operational  systems.  As 
a  result  of  many  years  of  effort  to  establish 
standardization  and  stability  in  the  lexicon  of  data 
fusion,  the  definition  of  many  terms  is  slowly 
achieving  consensus  across  the  diversified 
application  community.  Problem-specific  nuances 
and  shading  in  these  definitions  remain  but 
agreement  on  a  meaningful  subset  of  terms  does 
seem  to  exist. 

Data  Fusion  (DF)  is  an  adaptive  information 
process  that  continuously  transforms  the  available 
data  and  information  into  richer  information, 
through  continuous  refinement  of  hypotheses  or 
inferences  about  real-world  events,  to  achieve 
refined  (and  potentially  optimal)  kinematic  and 
identity  estimates  of  individual  objects,  and 
complete  and  timely  assessments  of  current  and 
potential  future  situations  and  threats  (i.e., 
contextual  reasoning),  and  their  significance  in  the 
context  of  operational  settings.  The  process  is  also 
characterized  by  continuous  refinements  of  its 
estimates  and  assessments,  and  by  evaluation  of  the 
need  for  additional  data  and  information  sources,  or 
modification  of  the  process  itself,  to  achieve 
improved  results.  Given  these  considerations,  a 
complete  data  fusion  system  can  typically  be 
decomposed  into  four  levels  (Refs.  2-3): 

Level  1  -  Multi-Source  Data  Fusion  (MSDF); 

Level  2  -  Situation  Assessment  (SA); 


29-3 


Level  3  -  Threat  Assessment  (TA);  and, 

Level  4  -  Process  Refinement  Through 
Resource  Management  (RM). 

Each  succeeding  level  of  data  fusion  processing 
deals  with  a  higher  level  of  abstraction.  Level  1  data 
fusion  uses  mostly  numerical,  statistical  analysis 
methods,  while  levels  2,  3  and  4  data  fusion  use 
mostly  symbolic,  Artificial  Intelligence  (AI) 
methods. 

Multi-Source  Data  Fusion  (Level  1) 

Multi-source  data  fusion  (MSDF)  is  concerned 
solely  with  individual  objects,  first  in  associating 
the  sensor  outputs  with  specific  known  objects  or 
using  them  to  initiate  new  objects.  Level  1 
processing  uses  sensor  data  to  correctly  and  quickly 
derive  the  best  estimates  of  current  and  future 
positions  for  each  hypothesized  object.  In  addition, 
inferences  as  to  the  identity  of  the  objects  and  key 
attributes  of  the  objects  are  developed. 

Situation  Assessment  (Level  2) 

Based  on  incomplete  and  inaccurate  sets  of  data  and 
information,  situation  assessment  (SA)  is  devoted  to 


the  continuous  inference  of  statements  about  the 
hypothesized  objects  provided  by  the  lower  level 
data  fusion  function  in  order  to  derive  a  coherent, 
composite  tactical  picture  of  the  situation.  This 
picture  must  be  described  in  terms  of  groups  or 
organizations  of  objects  so  that  enemy  intent  can  be 
estimated  in  the  next  level  and  decisions  can  be 
made  by  decision  makers  about  how  to  use  war 
fighting  assets. 

Threat  Assessment  (Level  3) 

Threat  assessment  (TA)  is  focused  at  the  details 
necessary  for  decision  makers  to  reach  conclusions 
about  how  to  position  and  commit  the  friendly 
forces.  By  coupling  the  products  of  situation 
assessment  with  the  information  provided  by  a 
variety  of  technical  and  doctrinal  databases,  TA 
develops  and  interprets  a  threat  oriented  perspective 
of  the  data  to  estimate  the  enemy  capabilities  and 
lethality,  identify  threat  opportunities  in  terms  of 
the  ability  of  own  force  to  engage  the  enemy 
effectively,  estimate  enemy  intent  (i.e.,  provide 
indications  and  warnings  of  enemy  intentions),  and 
determine  levels  of  risk  and  danger. 


Data  Fusion  &  Resource  Management 
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FIG.  1  -  Overlap  between  the  data  fusion  and  resource  management  domains 
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Resource  Management  ( level  4  -  Process 

Refinement  through  Sensor  Management) 

Information  resource  management,  level  4 
processing,  closes  the  loop  by  first  examining  and 
prioritizing  what  is  unknown  in  the  context  of  the 
situation  and  threat  and  then  developing  options 
for  collecting  this  information  by  cueing  the 
appropriate  sensors  and  collection  sources. 
Resource  Management  refers  to  the  continuous 
process  of  planning,  coordinating  and  directing 


the  use  of  the  ship  or  force  resources  to  counter  the 
threat.  It  is  concerned  with  issues  of  both  command 
and  control.  Note  that  resource  management  in  the 
context  of  level  4  fusion  is  mainly  concerned  with  the 
information  gathering  process  refinement  (i.e.,  sensor 
management).  However,  the  overall  domain  of 
resource  management  also  encompasses  the 
management  of  weapon  systems  and  other  resources. 
Fig.l  illustrates  the  overlap  between  the  data  fusion 
and  resource  management  domains. 


FIG.  2  -  A  Generic  MSDF  Model 


3.1  A  generic  MSDF  model 

Figure  2  shows  a  generic  MSDF  model  where  the 
key  functions  are  identified.  The  processing  can  be 
divided  into  blocks  such  as: 

•  data  alignment  (spatial,  time), 

•  data  association, 

•  target  kinematic  data  fusion, 

•  target  identity  data  fusion, 


•  track  management  process, 

•  cluster  management  process, 

•  input  data  preparation, 

•  track  database, 

•  fusion  configuration  monitoring  and  control. 

A  detailed  description  of  all  these  processes  is  provided 
in  Ref.  5. 
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In  any  MSDF  system,  sensor  data  alignment  in 
time  and  space  must  take  place  before  any  fusion 
can  be  performed.  Navigation  data  is  used  to 
estimate  and  remove  the  effects  of  sensor  motion 
from  the  received  data.  The  functions  of  data 
association  (labeling  measurements  from  different 
origins  and/or  sensors,  at  different  times,  that 
correspond  to  the  same  object  or  feature)  and  data 
fusion  (combining  measurements  from  different 
times  and/or  different  sensors)  are  also  required  in 
one  form  or  another  in  essentially  all  multiple 
sensor  fusion  applications:  data  association 
determines  what  information  should  be  fused,  the 
fusion  function  performs  the  fusion. 

In  addition  to  the  MSDF  system  which  detects, 
localizes,  and  identifies  targets,  the  local  sensor 
management,  on  the  basis  of  an  evolving  picture, 
and  under  the  command  of  the  overall  Command 
and  Control  (C2)  resource  management,  manages 
the  information  that  the  MSDF  might  receive  by 
pointing,  focusing,  maneuvering,  and  adaptively 
selecting  the  modalities  of  its  sensors  and  sensor 
platforms.  The  overall  C2  resource  management 


has  the  responsibility  of  first  examining  and 
prioritizing  what  is  unknown  in  the  context  of  the 
situation  and  threat  and  then  developing  options  for 
collecting  this  information  by  sending  priorities  to  the 
MSDF  local  sensor  management  function.  The 
interaction  of  the  local  sensor  management  and  the 
MSDF  function  is  done  through  the  fusion 
configuration,  monitoring  and  control  which  is 
responsible  for  the  initialization  of  the  MSDF  system, 
the  setup  and  the  adjustment  of  the  various  MSDF 
algorithm  parameters  to  control  the  quality  of  the 
MSDF  product. 

3.2  A  generic  STA  model 

Ref.  3  presents  a  generic  model  for  situation  and 
threat  assessment  influenced  by  the  human’s  mental 
processing.  The  model  evolved  from  a  three  level 
descriptive  model  of  situation  awareness,  where  the 
first  level  is  concerned  with  the  perception  of  the 
elements  in  the  environment,  the  second  level  is  about 
the  comprehension  of  the  current  situation  and  the  last 
level  deals  with  the  projection  of  the  future  states  of 
the  situation. 


Fig.  3  -  Generic  Model  for  STA 
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The  proposed  generic  model  for  STA  is  illustrated 
in  Fig.  3  and  consists  of  a  Perception  Refinement 
module,  a  Threat  Refinement  module,  a  Situation 
Interpretation  module,  a  Situation  Projection 
module,  a  Monitoring  module  and  a  Diagnosis 
module.  The  generic  model  has  access  to  a  priori 
knowledge  and  transient  information  and  its 
behavior  is  modulated  by  a  Meta  Controller  taking 
account  of  processing  priorities,  processing  time 
and  quality  of  the  information.  The  idea  here  is 
not  to  describe  all  these  entities  but  only  those 
which  can  be  used  to  complete  the  identity  data 
fusion  process  i.e  the  perception  refinement 
module.  We  refer  the  reader  to  Ref.  3  for  a 
detailed  description  of  the  STA  model  of  Fig.3. 

3.2.1  The  Perception  Refinement  module 

The  Perception  Refinement  module  is  composed 
of  three  sub-processes  :  the  Data  Refinement  (DR) 
sub-process,  the  Kinematic  Estimation  (KE)  sub¬ 
process  and  the  Behavior  Analysis  (BA)  sub¬ 
process.  The  idea  of  DR  is  to  refine  the  track  data 
(position,  identity)  already  generated  by  MSDF  to 
examine  the  data  set  for  incompleteness  and 
contradictions  and  to  establish  relationships 
among  the  entities  (in  terms  of  proximity, 
functionality  and  dependency)  with  the  help  of 
external  data  sources  if  necessary.  At  this  point, 
no  inferences  about  the  situation  is  done.  The  only 
results  obtained  from  this  sub-process  is 
perceptual  refinements. 

The  Kinematic  Estimation  sub-process  is  used  for 
the  computations  of  kinematic  information  (mean 
line  of  advance,  closest  point  of  approach  (CPA), 
time  of  flight  (TOF)...)  of  a  track  for  Threat 
Refinement  processing,  and  for  engageability 
calculations  done  in  Resource  Management.  An 
history  function,  which  records  the  positional 
tracking  and  identification  information  in  time,  is 
needed  to  accomplish  these  kinematic 
calculations.  By  means  of  this  history  function, 
the  recording  of  the  track  information  will  allow 
us  to  address  enemy  information  countermeasures 
(i.e.  information  warfare). 


Behavior  Analysis  (BA)  is  the  last  sub-process  under 
Situation  Refinement  and  is  used  to  analyze  the 
behavior  analysis  of  entities  and/or  cluster  in  order  to 
help  to  the  refinement  of  the  data  set.  Also,  BA 
becomes  a  preliminary  step  for  the  second  level  of  the 
situation  awareness  model  by  providing  processing 
cues  or  evidence  about  the  behavior  of  the  track  or 
cluster  to  the  Situation  Interpretation  module  for  the 
interpretation  and  understanding  of  the  current  tactical 
situation.  The  cues  are  obtained  from  the  analysis  of 
the  refined  information  (track  and  cluster’s  kinematic 
data  and  identification)  done  in  DR,  the  a  priori 
knowledge  and  from  transient  information  (i.e. 
electronic  emissions  from  ESM,  the  datalink, 
participating  units...).  BA  includes  functions,  such  as 
corridor  correlation,  maneuver/pattern  identification 
that  generate  the  required  cues  to  infer  about  the 
tactical  picture. 

4.  KINDS  OF  IDENTITY  INFORMATION 

In  a  maritime  environment,  various  surveillance 
systems,  electronic  intelligence  and  human 
observations  are  examples  of  information  sources 
available  to  the  commander.  Output  from  these 
sources  is  partitioned  according  to  the  type  of 
information  they  provide;  output  data  may  be 
characterized  as  either  positional  or  identity 
information.  Positional  information  represents  the 
dynamic  parameters  describing  the  movement 
associated  with  an  object  (contact).  This  generally 
includes  position,  velocity  and  acceleration.  Identity 
information  can  be  defined  as  declarations, 
propositions  or  statements  that  contribute  to  establish 
the  identity  of  an  object.  Equivalently,  identity 
information  may  be  seen  as  information  from  various 
sources  that  helps  in  distinguishing  one  object  from 
another.  Possible  values  for  identity  information  can 
span  the  range  from  sensor  signals,  to  attributes,  to 
identity  declarations,  as  depicted  in  Fig.  4. 

The  sensor  signals  represent  some  characteristics  of 
the  energy  sensed.  Attributes  such  as  size,  shape, 
degree  of  symmetry,  emitter  type,  etc.  are  inferred 
from  these  characteristics.  Identity  declarations 
specify  the  detected  object;  in  the  Canadian  Navy,  for 
example,  they  can  consist  of  a  general  classification  of 
which  the  observed  object  is  a  member  (surface 
combatant),  a  specific  type  of  ship  (frigate),  a  specific 
class  (City  Class)  or  a  unique  identity  (Ville  de 
Quebec).  Therefore  surface  combatant,  frigate,  City 
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Class  and  Ville  de  Quebec  are  all  examples  of 
identity  declarations.  Identity  declarations  can 
also  include  information  concerning  the  threat 
designation  of  an  object:  pending,  unknown, 
assumed  friend,  suspect,  friend,  neutral  or  hostile. 


Fig.  4  -  Identity  Information 

The  quantity  of  identity  declarations  available 
may  become  quite  imposing  and  diverse.  A 
potential  way  of  organizing  identity  declarations 
might  be  to  use  the  NATO  standards  for 
representing  maritime  tactical  information 
(STANAG  4420,  Ref.  6).  These  standards  are  in 
the  form  of  charts  that  define  the  full  range  of 
tactical  information  required  by  the  operational 
user  at  the  command  level.  The  charts  were 
created  to  establish  the  basis  for  developing  a 
standardized  representation  of  spatially  displayed 
tactical  data  using  symbology  and  colour  for 
NATO  maritime  units.  Ref.  7  presents  an  adapted 
version  of  a  tactical  information  hierarchy  for 
surface  and  air  objects  respectively;  certain  levels 
and  entries  have  been  omitted  for  simplicity. 


The  names  in  the  various  boxes,  which  are  indicative 
of  the  taxonomy  used  in  STANAG  4420,  represent 
identity  declaration  entities.  These  entities  illustrate 
the  sort  of  identity  declarations  provided  by  various 
sources  which  must  be  combined  in  order  to  obtain  an 
estimate  of  an  object's  identity. 

In  addition,  the  identity  information  can  be 
characterized  by  being  local,  remote,  organic  and  non- 
organic.  The  notion  of  local  and  remote  are  self- 
explaining  but  the  organic  and  non-organic  require  to 
be  defined. 

Organic  Information 

Data  collected  and  managed  by  assets  under 
Commanding  Officer’s  direct  control  can  be  defined 
or  characterized  as  organic  information.  Organic 
information  must  be  sufficiently  timely  and  accurate 
to  be  used  in  real-time,  responsive  systems. 
Consequently,  it  can  be  used  to  produce  a  local 
tactical  picture  related  to  the  situation.  Note  that 
remote  sources  can  be  organic  and  local  sources  can 
be  non-organic.  Here  are  examples  of  organic 
information:  radars,  datalink  (e.g.  Link-11),  ESM, 
weather,  ...etc. 

Non-Organic  Information 

Data  collected  by  agents  not  under  CO  direct  control, 
is  referred  as  non-organic  information.  Non-organic 
information  is  less  timely,  reduced  in  accuracy, 
differently  structured  and  has  differing  identification 
confidence  levels.  Transient  information  such  as 
intelligence  reports  or  PU’s  information  is  non- 
organic  as  means  to  support  the  various  processes 
providing  the  commander  at  sea  to  gain  a  level  of 
situational  awareness. 

Other  Type  of  Information 

Another  source  of  data  is  the  human  contribution  to 
the  identification  process  through  a  Human  Computer 
Interface  (HCI).  Human  can  participate  actively  by 
providing  inputs,  based  on  his  experience,  his  training 
and  his  awareness  of  the  situation  within  the 
identification  process.  The  precise  nature  of  this 
human-computer  interaction  remains  to  be 
determined. 
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A  Priori  Knowled.ee 

A  Priori  Knowledge  contains  static  information  as 
means  to  support  the  various  processes  providing 
the  decider  to  gain  a  level  of  situational 
awareness.  That  knowledge  is  a  component  within 
the  STA  model  as  opposed  to  inputs  derived  from 
external  sources.  A  Priori  Knowledge  is  used  by 
the  Perception  Refinement  module  to  refine 
identity  information.  Here  are  some  examples  of  a 
priori  knowledge  source:  social  and  political, 
geographical,  platforms  characteristics,  mission 
guidelines,  weapons  characteristics,  corridor  and 
flight  paths,  EM  characteristics,  lethality,  emitter 
characteristics,  doctrines,  ...etc. 

5.  ARCHITECTURES  FOR  IDENTITY 
FUSION 

One  of  the  key  issues  in  developing  a  MultiSource 
Data  Fusion  (MSDF)  system  is  the  question  of 
where  in  the  data  flow  to  actually  combine  or  fuse 
the  data.  The  MSDF  architecture  is  an  important 
issue  since  the  benefits  are  different  depending  on 
the  way  the  sensor  or  other  source  data  are 
combined.  There  are  four  broad  alternatives  to 
fusing  positional  information:  at  the  signal  level, 
at  the  contact  (plot)  level,  at  the  track  level  and 
finally  a  hybrid  approach  which  allows  fusion  of 
either  contact  or  track  data.  The  benefits  of  theses 
architectures  for  positional  fusion  have  been 
extensively  analyzed  in  (Ref.  8).  For  identity 
fusion,  there  are  several  types  of  architectures 
which  can  be  used:  signal  level  fusion,  feature 
level  fusion  and  decision  level  fusion.  Fig.  5 
displays  the  three  fusion  architectures  providing 
the  structural  basis  for  fusing  identity  information 
in  the  case  of  two  sensors.  It  should  be  noted  that 
the  output  from  each  level  of  processing 
represents  the  three  categories  of  identity 
information  depicted  in  Fig.  4. 

There  is  no  one  universal  architecture  which  is 
applicable  to  all  situations  or  applications.  The 
selection  of  the  MSDF  architecture  type  should  be 
aimed  at  optimizing  the  target  detection,  tracking 
and  identification  performance  required  for  a 
specific  platform  given  its  missions.  However,  the 
selection  is  also  constrained  by  the  technological 
capabilities  (both  hardware  and  software).  It 


depends  on  the  quality  of  the  sensors  being  fused,  the 
availability  of  computer  processing  power,  the 
bandwidth  of  the  available  data  transmission  paths, 
and  the  degree  to  which  operator  intervention  is 
required  or  desired. 


sensor  A  sensor  B 


FIG.  5  -  Identity  Information  Fusion  Architectures 

6.  THE  IDENTITY  DATA  FUSION  PROCESS 

In  Section  4  the  sources  of  information  are  classified 
as  organic  and  non-organic  with  the  possibility  that 
remote  sources  can  be  organic  and  local  sources  can 
be  non-organic.  With  the  increasing  use  of  the  remote 
sources  (e.g.  wide  area  sensors)  and  the  increasing  use 
of  information  that  has  already  been  fused,  there  is  a 
requirement  to  define  an  identity  information  fusion 
that  can  handle  both  organic  and  non-organic 
information  characterized  by  different  accuracy  and 
timeliness  (real-time,  near  real-time  and  non  real¬ 
time).  Fig.  6  shows  an  example  of  an  identity 
information  process  for  two  platforms  fusing  identity 
information  from  organic  and  non-organic  sources. 
When  remote  data  is  to  be  combined  with  organic 
fused  data,  the  relationship  between  the  two  sets  of 
data  should  be  indicated  to  avoid  the  problem  of  data 
duplication  or  data  looping.  In  Fig.  6,  we  propose  a 
distributed  approach  where  two  data  fusion  centers  are 
maintained  at  the  platform  level:  an  organic  data 
fusion  center  and  a  global  data  fusion  center. 

The  organic  data  fusion  center  produces  the  identity 
declarations  from  the  local  and  remote  organic  sources 
and  the  result  is  broadcast  to  all  other  platforms.  In 
Fig.  6,  the  organic  data  fusion  results  of  platform  #1 
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are  made  available  to  the  global  data  fusion  center 
of  platform  #2  and  the  same  way,  the  organic  data 
fusion  results  of  platform  #2  are  made  available  to 
platform  #1.  Two  track  databases  need  to  be 
maintained:  an  organic  and  a  global.  Fig.  6  also 
shows  the  remote  organic  sources.  For  example  in 
an  air  engagement,  some  sensors  may  be  slaved 
directly  to  some  platforms  for  Cooperative 
Engagement  Capability  (CEC)  in  which  selected 
data  is  transmitted  from  one  unit  to  another 
through  dedicated  communications. 

The  global  data  fusion  takes  place  at  the  platform 
as  well  but  the  results  are  made  available  only 


under  dedicated  broadcast  to  avoid  data  looping. 
Ideally,  if  each  fusion  center  utilizes  the  same 
algorithms  i.e.  to  produce  the  same  database  result 
when  the  same  data  are  provided  as  inputs,  each 
platform  will  share  the  same  global  picture.  However, 
this  is  not  likely  to  happen.  Similar  algorithms, 
procedures  and  format  are  most  likely  foreseeable. 
The  goal  here  is  for  each  platform  to  share  a  similar 
tactical  picture  where  the  differences  are  within  the 
range  of  the  uncertainty  of  the  overall  fusion  process. 
To  achieve  this  interoperability,  some  international 
standards  need  to  be  established  and  agreed  upon. 
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FIG.  6  -  Multiplatform  Organic  and  Non-Organic  Data  Fusion 


6.1  The  Organic  Identity  Information  Process 

In  the  organic  identity  information  fusion  process 
of  Fig.  7,  the  attribute  information  and  identity 
declarations  obtained  from  various  sources  are 
compared  with  a  Platform  Data  Base  (PDB) 
containing  all  the  possible  identity  values  that  the 
potential  target  may  take.  Each  record  of  this 


database  contains  information  related  to  the  measured 
sensor  attributes  and  identity  declarations.  Therefore, 
each  sensor's  attribute  information  is  translated  into  a 
subset  of  the  PDB  and  a  confidence  level  for  each 
subset  is  then  computed.  If  the  available  identity 
information  is  only  provided  by  self-contained  sensors 
producing  identity  declarations  in  an  autonomous 
fashion  or  by  remote  sources  offering  identity 
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might  be  quite  different  from  those  used  in  the 
organic  identification  process.  The  proposition 
construction  subprocess  is  dealing  only  with 
identity  declarations  and  then  does  not  require  the 
use  of  fuzzy  logic.  Since  the  identity  declarations 
are  hierarchically  structured,  fast  algorithms  such 
as  the  Shafer-Logan  (Ref.  7)  can  be  used.  Ref.  7 
also  discusses  how  the  hierarchy  affects  the 
decision  rules.  Unlike  the  organic  identification 
process,  where  only  organic  information  is  used  in 
the  refinement  of  the  identity  propositions,  the 
global  identification  process  makes  use  of  all  the 
available  information  to  refine  the  identity 
propositions. 

7.  CONCLUSION 

This  paper  proposed  an  architecture  that  can 
perform  the  fusion  of  organic  and  non-organic 
identity  data  through  the  use  of  statistical  analysis 
rooted  in  the  Dempster-Shafer  theory  of  evidence. 
The  architecture  has  the  benefit  to  provide  a 
solution  to  the  problem  of  data  duplication  or  data 
looping  when  remote  data  is  to  be  combined  with 
organic  fused  data.  The  distributed  approach 
consists  of  two  data  fusion  centers  maintained  at 
the  platform  level:  an  organic  data  fusion  center 
and  a  global  data  fusion  center. 
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PAPER  No:  29 

DISCUSSOR’S  NAME:  M.  Balci 

COMMENT/QUESTION: 

1 .  Is  there  any  difference  between  remote-organic  and  non-organic  information? 

2.  What  do  you  mean  by  one  single  identity  information  from  each  sensor? 

3.  Do  you  give  any  reference  for  defining  m  (0-)  values? 

AUTHOR/PRESENTER’S  REPLY: 

1 .  Yes,  remote  organic  information  means  that  the  decider  or  the  commander  has  control  over 
units  that  are  remote,  while  non-organic  means  that  the  decider  has  no  control  over  it. 

2.  If  there  is  more  than  one  attribute  given  by  a  particular  sensor,  we  propose  to  integrate 
those  attributes  in  a  single  proposition. 

3.  Up  until  now,  the  m(6)  values  have  been  determined  by  common  sense  and  are  rather  ad 
hoc.  We  plan  to  look  more  carefully  at  this  problem  in  our  future  work. 


DISCUSSOR’S  NAME:  M.  Desbois 

COMMENT/QUESTION: 

I  fully  agree  on  the  need  to  separate  organic  and  non-organic  data  to  reject  “incestual  fusion” 
(reinforcing  the  ID  by  using  your  own  information).  However,  among  the  ID  sources  you  have 
IDBO  (Identification  by  Origin)  and  the  Track  Behaviour  (altitude,  aspect,  heading,  ...). 

1 .  Do  you  process  these  sources. 

2.  If  yes,  would  you  classify  these  sources  as  organic  data? 

AUTHOR/PRESENTER’S  REPLY: 

1.  Yes,  we  could. 

2.  It  depends  on  the  application,  but  generally  speaking  if  the  source  delivering  that 
information  is  under  the  control  of  the  decider  I  would  classify  it  as  organic  data. 
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PAPER  No.  29 

DISCUSSOR’S  NAME:  E.  Schweicher 

COMMENT/QUESTION: 

Dans  les  menaces  requerant  une  reaction  rapide  incluez-vous  les  mines  derivantes  et  les  missiles 
a  vol  rasant  tels  que  le  missile  EXOCET? 

(Do  you  include  drifting  mines  and  sea-skimming  missiles  such  as  EXOCET  in  the  category  of 
threats  requiring  rapid  reaction?) 


AUTHOR/PRESENTER’S  REPLY: 

Absolument.  Pour  ce  qui  a  trait  aux  missiles  a  vol  rasant,  le  systeme  de  fusion  devrait  etre 
capable  d’ identifier  cette  menace  le  plus  tot  possible  et  de  deleguer  a  un  sous-systeme 
automatique  le  soin  de  la  pister  avec  grande  precision. 

(Absolutely.  In  so  far  as  concerns  sea-skimming  missiles,  the  fusion  system  should  be  capable  of 
identifying  this  threat  as  soon  as  possible  and  assigning  the  task  of  high  precision  tracking  to  an 
automatic  sub-system.) 
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SUMMARY 

The  opportunity  to  process  several  images  being  acquired 
simultaneously  over  the  same  landscape  in  different  spectral 
bands  provides  a  better  perception  of  the  scene  using  data 
fusion  methods.  This  new  method,  classifying  the  image  in 
two  types  (target  and  background)  using  the  radiometric 
properties  of  the  surface,  is  particulary  powerful  for  the 
detection  of  non  resolved  objects.  The  fusion  method  is 
defined  by  using  the  normal  probability  density  hypothesis  to 
estimate  the  ponderation  of  each  initial  image.  The 
classification  is  done  by  using  a  threshold  where  analytic 
expression  depends  on  the  type  of  the  "a  priori"  knowledge. 
The  method  is  then  applied  on  an  image  acquired  with  the 
hypcrspectral  scanner  sensor  :  AVIRIS.  The  chosen  target  is  a 
non  resolved  road  in  one  dimension.  Different  criteria  are  used 
to  evaluate  the  performance  of  the  method  :  statistical 
properties,  good  classification  percentage,  false  alarm  and  non 
detection  errors.  We  analyze  the  "a  priori"  probability 
uncertainty  on  the  image  obtained  after  fusion. 

LIST  OF  SYMBOLS 

i :  the  index  of  the  considered  band  (i  =  1. ....  n), 

j,  j’:  the  current  index  of  the  landscape;  j  or  j’  =  t  for  the 

target  and  j  or  j’  =  b  for  the  background, 

n  :  number  of  channels  to  be  combined. 

x  :  pixel  spectral  signature  of  component  x1  ( dim  x  =  n  ), 

D(d , ,  d  b )  :  decisions  made  about  the  pixel  nature, 
dt  shows  that  pixel  is  belonging  to  the  target, 
db  shows  that  pixel  is  belonging  to  the  background, 

W(t.b)  :  the  two  possible  classes, 

Cjj  =  c  (d  j  /  j1 )  :  the  cost  combined  with  the  decision  dj  when 

the  nature’s  statement  is  j’, 

R  :  the  conditional  risk, 

P(error) ;  the  mean  error  probability, 

p(x)  :  the  density  probability  of  the  x  vector, 

P(j)  :  the  "a  priori"  probability  that  a  pixel  belongs  to  the 
database  j, 

P(j/x)  :  the  "a  posteriori"  probability  that  the  nature’s 
statement  is  j  with  x  well-known. 

p  (x  /  j)  :  the  density  probability  of  x  according  to  the  j 
nature’s  statement. 

Ij  ;  the  covariance  matrix  of  the  database  j, 

rh  j  :  the  mean  vector  of  the  database  j  of  component  m '• , 

(7j  ;  the  standard  deviation  of  database  j  from  the  band  i, 

g(x)  :  the  combination  rule, 

a  :  the  ponderation  vector, 

thj  :  the  database  mean  of  the  fusionned  image, 

5j  :  the  database  standard  deviation  of  the  fusionned  image, 
s  :  the  threshold. 

R, :  the  area  associated  to  the  j  database. 


nj  ;  the  pixel  number  of  the  database  j  whose  level  values 
belong  to  the  Rj  area, 

Nj  :  the  total  pixel  number  on  the  j  database, 

XS; :  the  i  spectral  band  of  the  SPOT  instrument. 

LP  :  the  equivalent  radiance  at  a  P  point, 

SXsi :  the  spectral  sensibility  of  the  XSi  sensor. 

1.  INTRODUCTION 

The  large  quantities  of  data  provided  by  satellite  and  airborne 
sensors  lead  to  the  consideration  of  efficient  methods  as 
fusion  that  combine  several  information  from  multiple  sensors 
and  significantly  increase  scene  perception.  The  data  diversity 
and  the  different  processing  level  imply  the  development  of 
several  fusion  techniques. 

In  the  image  domain,  Waltz  [1]  distinguishes  three  levels 
according  to  the  information  abstraction  degree  ; 

•  The  fusion  pixel,  that  is  close  to  raw  observations, 

handles  with  pixel  level  information  that  are  directly 

provided  by  sensors. 

•  The  numerical  fusion  integrates  attribute  data 

obtained  after  a  segmentation  stage. 

•  The  symbolic  fusion  combines  information  made  up 

by  hypothesis  or  suggestions. 

Our  study  is  in  the  context  of  pixel  fusion.  Now,  the  various 
fusion  methods  can  gather  in  two  classes  according  to  the 
used  combination  rule:  weighted  means  or  weighted  means 
ratio.  So,  Gonzales  [2]  defines  the  multispectral  image 
ponderation  from  the  principal  components  analysis  (PCA). 
Likewise,  Arigides  [3]  and  Stocker  [4,  5]  use  the  bi-spectral 
difference  in  infrared  domain  in  order  to  remove  the  artificial 
targets  background.  According  to  the  available  "a  priori" 
knowledge  nature,  various  criteria  are  used  to  elaborate  the 
multispectral  observations  ponderation.  The  correlation  matrix 
allows  to  merge  information  by  minimizing  the  background 
noise  power.  The  further  knowledge  of  pixel  signature  tends 
towards  the  maximum  contrast  between  target  and 
background  and  so  provides  the  best  results  in  signal  to  noise 
ratio  [4],  Methods  based  on  weighted  means  ratio  generally 
use  the  physical  properties  of  the  studied  target.  An 
application  is  the  vegetation  detection  by  the  vegetation  index 
[6]  defined  as  a  normalized  spectral  contrast.  However,  these 
performance  are  sensitive  to  external  parameters  :  precision  of 
the  sensor  radiometric  calibration,  atmospheric  and  geometric 
viewing  conditions.  Various  authors  have  tried  to  minimize 
their  impact  either  by  introducing  adjustment  parameters  ([7]) 
or  by  developing  a  non  linear  expression  of  this  index  ([8]).  In 
the  infrared  domain,  another  application  [9,10]  is  the 
detection  of  buried  minefields  whose  principle  consists  in 
dissociating  the  reflexion  and  emission  phenomenon  from 
images  acquired  in  the  3-5  pm  and  10-12  pm  bands  by 
exploiting  the  radiometric  behaviour  difference  between 
natural  and  artificial  surfaces. 

Therefore,  fusion  allows  to  improve  classification  either  by 
directly  analyzing  the  fusionned  image  or  by  exploiting  the 
fusion  results  at  the  beginning  of  the  classification  algorithm  in 
order  to  increase  performance.  Most  of  these  techniques  are 
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based  on  a  knowledge  base  whose  performance  is  directly 
dependent  of  its  quality.  Whatever  the  fusion  method  used, 
except  the  principal  components  analysis,  the  effort  is  made  to 
optimize  the  classification  algorithm  rather  than  analyze  the 
spectral  bands  choice  or  the  relative  importance  given  for  each 
band.  Moreover,  the  performance  obtained  by  the  fusionned 
image  is  not  directly  evaluated  on  the  image  but  is 
determinated  only  on  the  classification  results. 

We  propose  a  new  data  fusion  method  at  pixel  level  applied  to 
estimate  multispectral  images  ponderation  coefficients  for 
supervised  classification  of  non-resolved  objects  and  optimize 
target  detection  on  background.  The  bayesian  approach, 
described  in  §  2,  is  used  to  select  the  channel  combination 
rule,  that  minimizes  the  global  risk.  Therefore,  a  channel 
combination  rule  is  proposed  by  estimating  the  probability 
density  functions  of  the  target  and  background.  Then 
thresholds  are  defined  and  applied  to  the  initial  and  fusionned 
images.  The  originality  of  our  work  lies  in  the  results 
assessment  that  is  directly  obtained  on  the  image.  Moreover, 
different  criteria  are  defined  (§  2)  to  evaluate  the  classification 
performance.  Our  method  is  applied  to  multispectral  images 
taken  by  the  AVIRIS  hyperspectral  imaging  spectrometer  for 
road  detection.  This  last  part  allows  the  comparison  of  results 
obtained  according  to  the  "a  priori"  knowledge. 

2.  METHOD  DESCRIPTION 

The  pixel  fusion  method  is  based  on  a  probabilistic  approach 
that  supposes  the  probability  laws  of  each  class  known. 

2.1.  Pixel  fusion 

2.1.1.  Principle 


•  ctt  =cb5  =0  :  the  cost  to  make  the  decision  dj  when  the 

nature  statement  j  is  minimum.  So  the  cost  to  make  a  right 
decision  is  non-existent. 

•  ctb=cbt=l:  the  cost  to  make  the  wrong  decision  to 

allocate  the  pixel  to  a  class  is  maximum. 

The  decision  rule  that  makes  the  global  risk  be  minimum 

,  .  ,  p(x/ 1)  .  P(b) 

value  is  then  :  — - )  — — 

p(x/b)  P(t) 


DUDA  [11]  shows  that  this  rule  makes  the  mean  error 
probability  be  the  minimum  value  : 

+oo 

P(error)  =  J  P(error,x)dx 

Then,  we  try  to  minimize  the  mean  error  probability. 


2.1.2.  Bayesian  method 

When  the  probability  densities  p(x,j)  of  target  and 
background  follow  a  gaussian  law  N(m,Z)  and  the  "a  priori" 
probabilities  are  determinated,  a  function  g  of  the  x  vector, 
that  separates  target  from  background,  can  be  calculated.  Let 
p(x  /  j)  =  N(fn j  ,Z  j) ,  the  decision  aile  becomes  : 

^target  <=>  p(x/  t).P(t)  >  p(x/  b).P(b) 

Using  the  normal  density  probability  expression,  we  obtain  : 
dtarget  <=>  - -  m , ] ‘ Z  t  ^  [x -  m t ]+ Log(P(t)) 

-j^L°g(de  t(Z  t)|  )- j^[x-mb]cZ  b  _1  [x-inb]  + 
Log(P(b))  -  Y2  Log(de  (Z  b )) 


The  target  detection  is  obtained  by  minimizing  the  global  risk. 
The  conditional  risk  is  the  mean  cost  combined  with  the  dj 
decision  when  the  multispectral  observation  is  x  : 

R(d,/x)=  £c(dj  /  j’)P(j7x)  (1) 

j-b.t 

Let  D:  x— >D(x)=dt  or  db  be  the  decision  rule  that 
points  out  the  decision  to  make  when  the  spectral  signature  is 
x  .  Thus,  the  global  risk  R  designates  the  mean  cost  combined 
with  a  decision  rule  :  R  =  j  R(D(x)/  x)  p(x)dx 

The  bayesian  detection  requires  that  the  applied  decision  rule 
minimizes  the  global  risk.  Therefore,  a  pixel  x  belongs  to  a 
target  if  and  only  if  the  conditional  risk  is  minimum,  then  : 

R(dt  /  x) <  R(db  lx) 

Using  (1),  we  obtain  : 

(c,b  -cbb  )P(b  /  x)  <  (cbt  —  ctt  )P(t  /  x) 

We  require  that  the  error  cost  is  higher  than  the  cost  in  the 
absence  of  error:  (ctb -cbb) )  0  and  (cb, -c„))0. 

The  Bayes’  rule  allows  us  to  express  the  "a  posteriori" 
probabilities  as  a  function  of  the  "a  priori"  probabilities  and 
the  conditional  densities: 

P(x) 

Consequently,  the  decision  rule  becomes  : 

D.  . .  ,  ,  .  .  p(x/t)  .  ctb  -cbb  P(b) 

Pixel  belongs  to  target  <=>  — - -  > - — - 

p(x/b)  cbt-ctt  P(t) 

Note:  If  the  cost  functions  are  unknown,  the  expression 
0<  Cj  j  <  1  is  used  to  determine  the  associated  values  : 


that  we  rewrite  as  : 

^target  »  g(x) )  a',-a'b  (2) 

with  : 

g(x)  =  xl(At  -  Ab  )x  +  (a,  -ab )  lx 


a'j =  j1*,  +  ~Log(de  t(Z  j ))  —  Log(P( j)) 

According  to  equation  (2),  g(x)  designates  a  quadratic 
function  of  x .  In  other  words,  the  spectral  signatures 
combination  of  every  pixel  must  be  quadratic  so  that  the  error 
mean  probability  corresponds  to  the  minimum  value. 

Note  :  When  the  normal  density  probability  hypothesis  is 
taken  into  account  and  the  target  and  background  covariance 

matrix  are  supposed  equal  t  =  Z  b  =  Z  ^  ,  we  obtain 

At  =  Ab . 

The  decision  rule  that  minimizes  the  error  mean  probability  is 
given  by  : 

a  1  x  )  a't  -a'b  with  a  =  X  _1  (m  t  -  m  b ) 

Thus,  the  fusionned  image  is  now  obtained  according  to  the 
linear  combination  rule  g(x)  =  a  lx  . 

2.2.  Pixel  classification  for  bayesian  method 

The  image  classification  in  two  objects  (target,  background)  is 
obtained  by  applying  a  threshold  on  this  image.  The 
classification  is  carried  out  at  2  levels.  The  first  level 
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corresponds  to  the  initial  image  thresholding  and  the  second 
level  is  the  fusionned  image  thresholding.  These  two 
classification  levels  results  allow  us  to  assess  the  fusion 
performance  in  relation  to  the  initial  images  classification. 
Threshold  is  chosen  according  to  the  classification  level. 


2.2.1.  Initial  images 


We  consider  that  images  are  independent  in  the  sense  that  a 
threshold  is  computed  for  every  initial  image.  Thus  for  every 
initial  image,  the  databases  are  modelled  by  a  one  dimension 
gaussian  density  probability. 

The  band  i  hypothesis  is  the  following  : 

Pi(x*  /  j)  =  N(m-,CTj) . 

The  decision  rule  can  be  written  as  : 

dtarget  <=>  Pi(x‘ /  t).P(t)  )  p;(x‘ /  b).P(b)  . 

To  simplify  our  notations,  the  sign  i  identifying  to  the  spectral 
band  will  be  deleted.  The  above  calculations  have  to  be  made 
for  every  spectral  band. 

The  density  formula  allows  us  to  simplify  the  decision  rule 
expression  : 

1  r  1  +  Log(P(t))-Log(at ) ) 


d, 


arget 


X  -  mb  +  Log(P(b))  -Log(ob ) 


_ 1_ 

Equality  is  obtained  when  the  x  value  of  the  pixel  P  is  equal  to 
the  classification  s  threshold  which  allows  to  separe  the  target 
from  the  background  databases. 

Equality  leads  to  this  quadratic  equation  : 

1,1  1  mt  m5  mr  mi 

-x  (—  )  +  x(— - -)- 

2  °b  at  °t  °b 

T  ,ot  •  P(b)  A 

Los(— - )  =  0 

obP(t) 

The  equation  (3)  can  be  reduced  to  a  a  first  degree  equation  if 
the  coefficient  of  x2  is  non-existent : 


2 -or  2- cl 


(3) 


,  1 
a  =  — 
2 


=  0  <=>  ot  =  ob  =  o 


So,  for  equal  target  and  background  databases  standard 
deviations,  there  are  various  cases  : 

•  If  mt  =  mb  and  P(t)  =  P(b),  the  two  density 

probabilities  are  superimposed  along  with  equal  "a  priori" 
probabilities,  then  there  Is  an  infinite  number  of  the  equation 
(3)  solutions  nevertheless  none  could  separate  the  target  from 
the  background. 

If  P(t)  *  P(b) ,  the  (3)  equation  doesn’t  have  any  solution 
and  another  threshold  type,  called  barvcentric  threshold,  is 
introduced  : 

o  b  m,  P(b)  +  CT  t  mb  P(t) 

S  o,P(t)  +  obP(b) 

•  If  m  t  *  m  b  ,  the  equation  (3)  solution  leads  to  one 
threshold  value,  called  gaussian  threshold  : 


1  , 

s  =  — (m,  + 1 
2 '  1 


+  - 


-Log 


'P(b) 

,P(t) 


111 1  _ulb 

When  ct,  *  ot,  the  (3)  equation  roots  number  depends  on  the 
discriminant  value  : 


1  ,11 

A  =  — ~ — — (mt  “  mb)"  +2  — - : 
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log 


qtP(b)l 
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We 


analyse  the  strictly  necessary  conditions  to  have  : 


A  =  0  «  mb  -mt  =  ±  2(ob -of)- Log 


otP(b) 

o„P(t) 

The  discriminant  cancels  out  if  and  only  if  one  of  the  two 
following  conditions  is  achieved  : 

So,  the  threshold  value  obtained  with  the  equation  (3)  is  : 

m1o|_-mbCT[ 

S  OO 

This  threshold  is  called  gaussian  threshold. 

.  ,  ,\  ( G.P(b) 

.  A)0»(mb-mt)'>2[ab-orj-Log|— ^ 

(ob)a,  and (a) 
b  1  P(t)  P(b) 

or 

(','>,J|’”dTw>TO)  (b) 

In  that  instance,  the  solution  (3)  leads  to  two  threshold  values 
smin  and  smax  called  gaussian  thresholds  : 


smax  = 


— b  +  Va" 


2a' 

m,  mb 


and 


-b-VA 

2a’ 


with 


b=2Iih - . 

°b 

Therefore,  the  decision  rule  is  :  dtarget  s  min(x(smax 

Two  further  conditions  which  depend  on  (a)  or  (b)  conditions 
are  obtained  : 

(c)  ob)crt  and  cj^mb<Gbmt  if  condition  (a)  is  respected. 

(d)  at)(Tb  and  0^mb)a^mt  if  condition  (b)  is  respected. 

ocP(b) 


A{0  <=>  (mb -mt)2(2(a^ -Ojl-Log 


obP(t) 

The  equation  (3)  doesn’t  supplied  threshold  value.  Then  the 
barvcentric  threshold  is  chosen  in  order  to  classify  image. 


2.2.2.  Fusionned  image 

For  normal  density  probability  hypothesis,  a  threshold  can 
directly  be  determinated  by  the  bayesian  inference  (cf.  §  2.1.2) 
and  the  density  probability  formula. 

In  fact,  the  decision  rule  leads  to  : 

d  target  «  g(x)>a't-a'b 

with  a'j  =  •im  jZ  T1  ni  j  +yLog(de  t(Z  j)j  -  Log(P(j)) 

The  threshold,  called  bavesian  threshold,  is  due  to  the  decision 
rule  :  s  =  a't-a'b  . 


2.3.  Evaluation  criteria 


The  images  fusion  use  is  justified  if  the  result  image 
performance  is  better  than  in  the  best  initial  images.  This 
contribution  is  evaluated  with  several  criteria  depending  on 
the  selected  processings.  These  criteria  are  all  defined  from 
the  two  databases. 

The  first  and  second  order  statistical  characteristics  analysis  of 
the  initial  and  fusionned  images  leads  to  make  up  two 
performance  evaluation  criteria. 

The  normalized  deviation  (in  %)  between  the  target  and 


background  mean  levels,  noted  Am  ,  is  : 


Am  = 


Ft  -mb| 
(mt  +mb)/2 


•100 
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with  m  j  mean  equals  to  the  mean  fh  j  in  the  fusionned  image 

or  to  the  mean  mj  in  the  initial  image  i. 

This  criterion  is  comparable  with  a  contrast  measurement 
between  target  and  background  for  a  given  initial  band  or  a 
given  fusionned  image. 

The  class  overlap  in  the  fusionned  or  initial  image,  written  Ar, 
is  defined  with  the  maximum  and  minimum  levels  of  the  target 
and  background  databases. 

For  instance,  if  min(xb )  <  min(xt)  <  Max(xh)  <  Max(xt) 
then  the  class  overlap  percentage  is  written  : 

Ar  =  Max(xb)  -  min(x, )  1Q() 

Max(xt )-  min(xb ) 

This  criterion  gives  information  about  the  position  of  the 
target  database  level  variation  area  compared  to  the 
background  one. 

Three  further  criteria  are  drawn  up  from  classified  initial  and 
fusionned  images  in  addition  to  a  classified  images  visual 
analysis.  These  criteria  are  defined  from  target  and 
background  databases  analysis. 

Let  us  consider  mt{mb  .  The  area  Rt  corresponds  to  the  part 
situated  before  the  classification  threshold  .?  and  the  area  Rb  is 
the  one  situated  after  the  threshold. 

The  good  classification  rate,  BC,  is  obtained  by  (in  %) : 

BC  =  ~”t+nb  •100 

N,  +Nb 

The  non-detection  error,  noted  E„d,  corresponds  to  the  case 
where  the  pixel  is  in  the  Rb  area  but  belongs  to  the  target 
class  : 


So,  higher  is  this  rate,  more  important  is  the  number  of  pixels 
classed  as  "background"  while  there  are  coming  from  target 
object. 

The  false  alarm  error,  noted  Ela.  is  obtained  by  computing  the 
pixels  number  in  the  R,  area  but  belonging  to  the  background  : 


These  criteria  will  be  used  afterwards  to  evaluate  the  fusion 
and  classification  pixel  performance. 

3.  DATA  DESCRIPTION 


All  the  spectral  manipulated  pixel  values  arc  expressed  in 
spectral  radiance  (W.m^.sr'1.  pm'1). 

The  images  (512  lines  by  614  samples)  used  in  this  study  were 
acquired  above  the  La  Crau  plain  on  the  18th  of  June  1997  at 
llh57  a.m.  This  site  is  a  plain  which  is  situated  at  50  km  in 
the  north-west  of  Marseille  and  diagonally  crossed  by  a 
highway  divided  with  a  grassy  central  reservation.  On  both 
sides  on  this  road,  there  are  some  orchards  in  the  North  and 
marshes  in  the  South.  The  rest  of  the  plain  is  composed  of 
stones  and  thick  bushes.  Some  scattered  clouds  (along  the 
image  right  border)  as  well  as  their  shade  appear  (Figure2). 
The  images  used  to  our  processing  simulate  the  SPOT 
instrument’s  three  spectral  bands  (Band  XSp  500  -  590  nm; 
XS2:  610  -  680  nm;  XS3:  790  -  890  nm)  and  are  reconstructed 
with  AVIRIS  hyperspectral  acquisitions. 

For  the  selected  spectral  band  XSj,  the  equivalent  radiance 
[14]  at  a  P  point  is  ; 


I  Lp(A.)Sxsi(k)dk 


LpfXSi)  ■ 


XS; 


J  Sxsi  t  ^  )dA, 

XSj 


inW.m'2.  sr'.pm’1 


Fusion  and  pixel  classification  will  be  carried  out  on  area  that 
are  extracted  from  these  different  images  XS  (Figure  2). 


3.2.  Databases 

A  minimum  "a  priori"  knowledge  of  the  target  and 
background  databases  is  required  in  order  to  have  a  reliable 
and  effective  pixel  fusion.  Our  method  is  applied  to  realize 
supervised  road  detection.  Then,  the  two  databases  are  built 
by  selecting  target  and  background  pixels  in  the  SPOT 
images. 

In  order  to  construct  the  target  database,  pixels,  the  most 
representative  of  the  radiances  variations  area,  are  taken  from 
different  parts  of  the  highway.  The  background  database 
(Figure  1)  is  taken  from  an  area  composed  of  various 
vegetation  types  (growed  fields,  orchards)  and  bare  soils 
(bushes,  stones). 


3.1.  Images 


Figure  1  :  Background  database  extracted  from  XS2 
image. 


Data  are  provided  by  the  AVIRIS  airborne  spectro-imager, 
covering  the  visible  and  the  near  infra-red  spectral  domain. 
AVIRIS  is  an  instrument  characterized  by  a  17  m  by  17  m 
ground  spatial  resolution  and  an  image  area  of  around  10  km 
by  up  to  21.5  km.  This  instrument  simultaneously  acquires 
images  in  224  narrow  spectral  bands  which  are  adjacent  10 
nm  wide  (+/-  2  nm)  and  covers  the  whole  solar  spectral 
domain  from  0.4  pm  to  2.5  pm  (Table  1)  [12,13], 


Spectral  characteristics 

Values  (nm) 

Spectral  domain 

400  a  2450 

Sample  interval 

10 

Channels  width 

10 

Calibration  accuracy 

<  2 

The  characteristics  of  the  databases  are  given  in  Table  2. 

We  compare  the  ratio  o/m  between  databases  for  a  given  band 
and  notice  that  the  background  database  dispersion  is  higher 
than  the  one  from  target  for  XS1  and  XS2  bands  (increasing 
approximately  of  76  %  from  target  to  background).  The  cj/m 
ratio  is  high  for  the  background  database  because  the 
radiances  are  scattered  owing  to  the  variety  of  the  surfaces 
which  make  up  this  database.  The  road  radiances  are  fairly 
uniform  because  road  is  tared. 

The  Am  deviation  between  the  target  and  background  mean 
radiances  increases  with  the  wavelength  (increasing  of  60  % 
from  XS1  to  XS2  and  of  70  %  from  XS2  to  XS3L  The  XS1 
and  XS2  bands  are  characterized  by  a  high  Ar  overlap.  On  the 
other  hand,  this  overlap  decreases  of  50  %  from  XS2  band  to 
XS3  but  remains  quite  considerable. 


Table  1  :  AVIRIS  spectral  characteristics. 
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XS1 

XS2 

XS3 

a/m 

Target 

9 

12 

10 

Background 

16 

21 

10 

Am 

12 

19 

32 

Ar 

75 

67 

38 

Table  2  :  Target  and  background  database  statistics 
determinated  on  initial  images  (%). 

4.  PIXEL  CLASSIFICATION  RESULTS 

We  consider  two  cases  :  the  "a  priori"  probabilities  of  target 
and  background  are  well-known  or  not. 


4.1.  "A  priori"  probabilities  well-known 

The  target  and  background  "a  priori"  probabilities  are  directly 
computed  from  the  initial  images:  P,  =  0.01  and  Pb  =  0.99. 


4.1.1.  Initial  images  pixel  classification 

Theoretical  and  visual  thresholds  that  are  applied  on  initial 
images  lead  to  a  image  classification  in  two  objects  :  target 
and  background.  The  visual  threshold  is  the  optimal  one  in  the 
sense  of  quality  image  criterion. 

The  best  results  are  obtained  in  band  XS3  and  are  display  in 
Figure  2. 


Theoretical  threshold 


Visual  threshold 


Figure  2  :  Image  XS3  classified  by  bayesian  method 
with  the  known  "a  priori"  probabilities. 

The  class  recovery  rates  and  the  mean  radiances  deviation 
obtained  in  the  3.2  paragraph  are  in  harmony  with  this  remark. 
In  fact,  the  best  classification  is  realized  when  threshold  is 
applied  on  the  XS3  band  because  for  this  band  the  rate  of 
radiances  which  are  identical  to  target  and  background  is 
lower  and  the  target  and  background  mean  radiances 
deviation  is  higher.  Nevertheless,  scattered  cloud  shades  (on 
right  border  image)  and  pixels  stemming  from  the  swampy 
area  (situated  at  the  left  bottom  of  image)  remain  classified  in 
target  in  the  XS3  band. 

The  search  for  visual  classification  threshold  leads  to  a 
different  value  from  the  one  provided  by  theory  (§  2.2).  For 
normal  density  probability  hypothesis,  only  the  XS3  image  can 


be  classified  by  gaussian  thresholds  (there  isn’t  solution  lor 
the  others  bands  cf.  §  2.2. 1.1. ).  For  the  XS3  band  classified  by 
visual  threshold,  only  some  pixels  delimiting  clouds  shades 
belong  to  the  target  class  while  for  theoretical  threshold  the 
whole  shades  were  detected  as  target  object.  However,  pixels 
belonging  to  the  road  beginning  are  obscured  by  trees  which 
are  located  on  the  road  border  or  their  shades  while  the  two 
road  ways  were  detected  on  the  XS3  image  classified  by 
visual  threshold;  what  does  not  perfectly  point  out  the  real 
landscape. 

The  XS1  and  XS2  bands  are  highly  correlated,  this  can  be 
shown  in  Table  3. 

Afterwards  the  evaluation  criteria  results  that  are  obtained  by 
classification  after  fusion  will  be  compared  with  results 
determinated  by  initial  XS3  image  and  gaussian  thresholds. 


Criterion 

E„d 

Efa 

BC 

XS1  Image 

43.0 

17.6 

81.7 

XS2  Image 

41.6 

17.0 

82.5 

XS3  Image 

43.7 

0 

98.9 

Table  3  :  Pixel  classification  performance  evaluation 
criteria  on  initial  images  with  the  well-known  "a 
priori"  probabilities  (%). 

4.1.2.  Pixel  classification  after  fusion  by  bayesian  method 

The  Am  deviation  strongly  increases  and  the  Ar  recovery  rate 
is  equal  to  zero  after  fusion  (Table  4). 


Criterion 

Am 

Ar 

E„d 

Efa 

BC 

Well-known  "a 
priori"  probabilities 

>100 

1 

9.9 

1 

99.1 

Table  4  :  Performance  evaluation  criteria  of  after 
fusion  classification  with  the  well-known  "a  priori" 
probabilities  (%). 


The  fusion  leads  to  a  target  from  background  radiances 
partition. 

The  fusionned  image  classification  is  obtained  by  the  bayesian 
and  visual  thresholds  (Figure  3). 


Theoretical  threshold 


Figure  3  :  Bayesian  method  and  the  known  "a  priori" 
probabilities. 

In  the  case  of  classification  by  theoretical  threshold,  the 
classified  image  visual  analysis  shows  that  field  pixels 
corresponding  to  the  darkest  levels,  clouds,  pixels  close  to 
road  are  detected  as  target  after  fusion. 

In  the  case  of  classification  by  visual  threshold,  the  road  is 
perfectly  detected.  While  some  cloud  shades  belong  to  the 
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target  class,  the  number  of  pixels  belonging  to  clouds  and 
classified  as  target  (on  the  right  border  at  the  top)  has  strongly 
decreased  and  even  some  shades  have  disappeared.  Moreover, 
the  central  platform  that  divides  the  highway  clearly  appears. 
The  visual  threshold  improves  the  classification  in  relation  to 
the  theoretical  one.  In  that  case  a  low  threshold  value 
deviation  creates  considerable  change  on  the  classified  image. 
The  visual  analysis  of  the  image  fusionned  and  the  reference 
image  classified  by  visual  threshold  (Figure  2)  shows  that 
fusion  improves  classification:  a  part  of  the  cloud  shade  is  no 
more  detected  as  target  but  belongs  to  background,  the  other 
part  classified  as  target  has  decreased.  This  classification 
improvement  provided  by  fusion  is  low  because  the  XS3  band 
is  high-performance  and  the  associated  others  bands  are 
mediocre. 

At  the  database  level,  the  non-detection  error  computed  with 
bayesian  threshold  strongly  decreases  (77  %)  according  to  the 
reference  image  result.  On  the  other  hand,  the  false  alarm 
error  and  the  good  classification  rate  arc  approximately  equal 
before  and  after  fusion. 

Threshold  value  is  dependent  on  the  database  choice.  The 
bayesian  and  visual  thresholds  lead  to  different  results:  the 
bayesian  threshold  allows  rough  road  detection  while  the 
visual  threshold  realize  a  more  efficient  detection. 

This  kind  of  fusion  can  lead  to  small  size  target  detection.  On 
the  fusionned  image  classified  by  visual  threshold,  the  grassy 
central  reservation  is  clearly  detected  in  the  middle  of  the 
road.  Every  way  including  central  reservation  width  is  equal 
to  the  spatial  resolution  instrument.  The  small  target  detection 
Is  thus  conceivable  for  fusion  with  normal  probability 
hypothesis  and  visual  threshold. 

4.1.3.  Note 

A  high  non-detection  error  En[t  rate  is  obtained  on  the  initial 
and  fusionned  classified  images. 

The  Enci  deviation  is  very  sensitive  owing  to  the  low  number  of 
pixels  forming  the  target  database.  Moreover  this  high  E„d 
value  is  due  to  the  target  database  construction.  In  fact,  the 
target  pixels  are  of  two  types:  those  consisting  only  of  road 
objet  called  "pure  pixels"  and  those  resulting  from  the 
combination  of  road  object  with  either  the  grassy  central 
reservation  or  the  road  borders  stemming  from  the 
background  class  called  "mixed  pixels".  At  the  time  of  the 
classification,  the  "mixed  pixels"  separated  by  the  central 
reservation  are  allocated  to  the  background  class  while  they 
were  chosen  at  the  beginning  for  target  database  construction. 

4.2.  "A  priori"  probabilities  unknown 

When  the  target  and  background  "a  priori"  probabilities  arc 
not  known,  they  are  chosen  equal  to  0.5  (equiprobability 
hypothesis). 

4.2.1.  Initial  images  pixel  classification 

The  visual  analysis  proves  that  the  best  results  are  obtained 
for  the  XS3  image  as  in  the  case  where  the  "a  priori" 
probabilities  are  well-known.  On  the  XS3  image  classified  by 
gaussian  thresholds  (Figure  4),  some  background  areas 
characterized  by  the  darkest  levels,  the  cloud  shades  and  some 
marsh  pixels  belong  to  the  target  class. 


Figure  4  :  Image  XS3  classified  by  bayesian  method 
with  the  unknown  "a  priori"  probabilities. 

The  visual  thresholds  are  independent  of  the  target  and 
background  "a  priori"  probabilities  values.  The  XS3  image 
visual  classification  leads  to  better  result  than  these  obtained 
by  gaussian  thresholds.  In  fact,  the  road  separation  is  perfectly 
defined  and  some  pixels  of  the  cloud  shades  border  belong  to 
the  target  class. 

By  comparing  results  obtained  for  every  bands  (Table  5),  we 
notice  that  the  non-detection  error  increases  of  75  7c  from 
XS 1  to  XS2  while  the  false  alarm  error  decreases  of  20  7c  due 
to  some  vegetation  pixel  and  clouds  shades  detected  as  road 
class.  The  XS1  and  XS2  good  classification  rates  are 
approximately  equal.  The  XS3  non-detection  error  is  the  same 
as  the  one  obtained  for  XS1.  The  false  alarm  error  strongly 
decreases  from  XS2  to  XS3  (approximately  70  7c)  and  the 
good  classification  rate  increases  of  8  7c.  These  results  prove 
also  that  the  best  classification  is  obtained  by  XS3  band. 


Method  bayesian  classification 

E„d 

Efa 

BC 

XSl 

4.2 

13.0 

87.2 

Initial 

images 

XS2 

7.0 

10.2 

89.9 

XS3 

Theoretical 

threshold 

4.9 

3.3 

96.7 

Visual 

threshold 

26 

0 

99.3 

Table  5  :  Performance  evaluation  criteria  of  the 
initial  images  with  the  unknown  "a  priori" 
probabilities. 

The  visual  comparison  of  the  XS3  image  classified  using 
equiprobability  hypothesis  (Figure  4)  with  the  XS3  image 
classified  by  gaussian  thresholds  and  well-known  "a  priori" 
probabilities  (Figure  3)  shows  that  the  classified  initial  images 
are  very  dependent  on  the  "a  priori"  probabilities  choice. 
When  the  probabilities  arc  known,  the  classification  is  better 
in  relation  to  the  case  where  they  are  unknown  and  arbitrarily 
fixed.  In  fact,  the  knowledge  accuracy  increasing  allows  to 
decrease  the  number  of  pixels  located  near  the  road,  marsh 
pixels  and  pixels  from  darkest  areas  that  are  misclassified. 

At  the  databases  level,  the  non-detection  error  decreases  while 
the  false  alarm  error  increases  for  the  XS3  image  and 
arbitrarily  fixed  "a  priori"  probabilities.  This  non-detection 
error  decreasing  is  linked  to  the  4.1.3  note.  In  fact,  for 
equiprohability  hypothesis,  the  classification  by  gaussian 
thresholds  leads  to  a  rough  road  detection.  However,  the  road 
detection  is  realised  at  the  pixel  level  when  the  visual 
threshold  is  used  or  when  the  "a  priori"  probabilities  are 
known. 
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Afterwards,  the  XS3  classified  image  will  be  considered  as 
reference  image  in  order  to  analyse  the  bayesian  fusion 
performance. 

4.2.2.  Pixel  classification  after  fusion 

The  fusionned  image  classification  is  obtained  by  the  bayesian 
threshold  and  the  visual  one  associated. 

The  overall  visual  analysis  of  the  classified  image  by  bayesian 
threshold  (Figure  5)  shows  that  pixels  located  near  the  road, 
some  field  pixels  and  clouds  are  allocated  to  the  road  class. 
However  pixels  stemming  from  swamp  that  belonged  to  target 
on  the  XS3  classified  image,  are  classified  as  background  on 
the  fusionned  image. 

In  the  case  of  classification  by  visual  threshold,  the  road  is 
perfectly  detected  and  the  central  reservation  dividing  the 
road  clearly  appears.  However,  some  cloud  shades  are 
allocated  to  the  target  class.  In  that  case,  the  visual  threshold 
improves  results  in  relation  to  bayesian  threshold. 


Figure  5  :  Bayesian  classification  and  the  unknown 
"a  priori"  probabilities. 

The  visual  analysis  of  the  fusionned  image  and  the  XS3 
reference  image  that  are  classified  by  visual  threshold  (Figure 
4)  shows  that  fusion  improves  classification:  none  dark 
background  area  is  classified  as  target,  and  the  cloud  shades 
(located  on  the  right  border  image  at  the  bottom)  disappear. 
Moreover,  pixels  located  on  the  road  beginning  are  obscured 
by  trees  or  their  shades  while  the  two  roads  ways  appear  on 
the  XS3  classified  image  before  fusion.  So,  the  classified 
fusionned  image  perfectly  represents  the  real  landscape. 

At  the  databases  level  (Table  6),  the  non-detection  error 
obtained  by  theoretical  threshold  and  by  XS3  reference  image 
is  the  same.  However,  the  false  alarm  error  increases  as  the 
good  classification  rate  decreases  after  fusion.  Fusion  does 
not  improve  classification  by  bayesian  threshold  results. 

In  the  visual  threshold  case,  fusion  increases  non-detection 
error  of  25  %  in  relation  to  the  XS3  image  value.  This 
increasing  proves  that  road  detection  is  achieved  at  the  pixel 
level  (cf.  §  4.1.3)  by  visual  threshold.  The  false  alarm  error 
and  good  classification  rate  don’t  change. 


Criterion 

Am 

Ar 

End 

Efa 

BC 

Theoretical  threshold 

>100 

1 

4.9 

4.4 

95.6 

Visual  threshold 

>100 

1 

43.7 

0 

99 

Table  6  :  Performance  evaluation  criteria  of  after 
fusion  classification  with  the  unknown  "a  priori" 
probabilities  (%). 

The  visual  classification  after  fusion  is  better  than  the  XS3 
band  one. 

The  visual  analysis  of  the  images  having  been  classified  by 
visual  threshold  (Figure  3  and  Figure  5)  shows  that  results  are 
similar  when  the  "a  priori"  probabilities  are  known  or  not 


known.  In  fact,  the  target  and  background  "a  priori" 
probabilities  don’t  take  place  in  the  bayesian  fusion  channels 
combination  rule  (cf.  §  2.1.2).  Therefore,  the  "a  priori" 
probabilities  choice  has  no  effect  on  the  classification  results 
obtained  by  visual  threshold. 

The  "a  priori"  probabilities  are  taken  into  account  in  order  to 
determine  the  theoretical  threshold.  By  comparing  Figure  3 
with  Figure  5,  we  notice  that  a  better  classification  is  obtained 
by  bayesian  threshold  when  the  "a  priori"  probabilities  are 
known.  In  fact,  for  the  equiprobability  hypothesis,  lots  of 
pixels  located  near  the  road  and  some  pixels  belonging  to  dark 
vegetation  area  are  classified  as  target. 

As  a  conclusion,  the  classification  results  are  closely 
dependent  on  the  threshold  choice  and  the  "a  priori" 
probabilities  knowledge. 

5.  CONCLUSIONS  AND  PERSPECTIVES 

The  optimal  low  level  fusion  leads  to  a  channel  combination 
rule  that  minimizes  the  global  risk  and  consequently  the  mean 
probability  error.  The  hayesian  approach  is  used  to  realize  this 
optimization.  It  allows  to  assess  hypothesis  from  independent 
sensor  measurements  when  the  target  and  background 
conditional  distribution  and  the  "a  priori"  probabilities  are 
known. 

The  pixel  fusion  aims  at  classifying  the  image  pixels  as 
different  objects  (in  that  instance  two  types).  The 
classification  is  obtained  by  applying  on  the  fusionned  or 
initial  image  theoretical  threshold  and  visual  threshold.  The 
visual  threshold  is  the  optimal  one  in  the  sense  of  quality 
image  criterion.  The  theoretical  threshold  depends  on  the 
classification  level  and  is  the  result  of  databases 
measurements.  The  classification  obtained  by  threshold 
depends  on  the  database  choice.  This  classification  method  is 
fast  and  efficient  at  the  repartition  of  pixels  to  classes  level. 

The  result  performance  evaluation  is  complete.  Several 
criteria  take  into  account  databases:  statistical  errors,  and  two 
criteria  providing  information  on  the  distributions  position 
independently  of  threshold.  A  further  qualitative  analysis  in 
image  term  completes  these  criteria. 

The  "a  priori"  probabilities  knowledge  effect  on  these 
classification  methods  is  analysed  by  considering  the  case 
where  they  are  unknown  and  arbitrarily  fixed. 

When  the  "a  priori"  probabilities  are  known,  results  show  that 
the  initial  image  classification  applied  to  road  detection 
depends  on  wavelength  and  threshold  choice.  In  fact,  the  best 
classified  image  before  fusion  is  obtained  by  the  XS3  band 
with  gaussian  thresholds  which  means  normal  density 
probability  hypothesis.  The  fusionned  image  classification 
depends  on  the  threshold  choice. In  fact,  a  low  threshold 
deviation  creates  important  changes  on  the  classified  image 
before  and  after  fusion.  Moreover,  visual  and  theoretical 
thresholds  are  different. 

The  pixel  fusion  that  takes  into  account  the  gaussian 
databases  probability  laws  and  the  visual  threshold  allows  to 
improve  classification  in  relation  to  XS3  image  results.  The 
fusion  efficiency  for  pixel  classification  is  here  weakly 
highlighted  because  the  XS3  band  that  provides  a  good 
classification  is  combined  with  the  other  bands  that  provide  a 
bad  classification  in  order  to  obtain  the  fusionned  image. 

The  effect  of  target  and  background  "a  priori"  probabilities 
values  is  analysed  by  considering  the  case  where  these 
probabilities  are  unknown.  When  the  classification  is  obtained 
by  visual  threshold,  classification  before  and  after  fusion  are 
independent  on  the  "a  priori"  probabilities  choice.  However 
classification  by  theoretical  threshold  is  better  when  the  "a 
priori"  probabilities  are  known. 

The  results  lead  to  introduce  further  spectral  bands  in  order  to 
improve  the  fusion  performance. 
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These  classification  methods  will  be  used  to  detect  n  objects 
(n  >  2)  on  an  image  thanks  to  their  simplicity,  quickness  and 
efficiency.  Consequently,  the  probability  densities  of  the  main 
kind  of  surfaces  withdraw  from  SPOT  images  are  at  the 
present  time  analysed  and  modelised. 

From  the  fusionned  image  by  bayesian  method,  that  allows  the 
more  efficient  classification  according  to  our  results, 
classification  can  be  realized  by  two  kind  of  thresholds:  the 
theoretical  threshold  following  from  the  analytical  resolution 
and  the  visual  threshold  that  provides  the  best  visual  result.  It 
is  difficult  to  evaluate  the  visual  threshold  because  this 
threshold  is  obtained  from  a  subjective  criterion  but  it  is 
independent  of  the  target  and  background  "a  priori" 
probabilities  values.  On  the  other  hand,  the  theoretical 
threshold  depends  on  the  "a  priori"  probabilities  knowledge. 
But  it  is  very  difficult  to  obtain  an  "a  priori"  probability  set 
that  perfectly  represents  reality.  A  more  accurate  analysis  of 
the  "a  priori"  probabilities  and  visual  threshold  choices  will  be 
later  developed  by  introducing  the  uncertainty  theory.  In  fact, 
this  theory  allows  to  exploit  the  physical  knowledge  and  take 
into  account  the  more  accurate  as  possible  information. 

The  probability  theory  associated  to  the  bayesian  decision 
theory  is  a  fusion  method  widely  used  also  as  comparison  base 
to  the  other  models  [15].  However,  this  theory  doesn’t  take 
into  account  the  more  or  less  partial  physical  knowledge.  This 
is  a  further  reason  to  deal  the  bayesian  inference  to  the 
processings  of  uncertain  values  that  are  provided  by  the  non 
exclusive  hypothesis  [16]. 
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Comment  se  passe  E  entree  des  donnees  issues  des  senseurs  dans  la  base  lorsque  ces  donnees  ne 
sont  pas  independentes  entre  elles? 

(How  is  sensor  data  input  to  the  database  when  this  data  is  not  differentiated?) 
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(The  model  proposed  allows  for  input  of  both  independent  indices  and  partially  or  totally 
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1.  SUMMARY 

Although  the  potential  of  sensor  and  data  fusion  as  a 
way  to  improve  situation  awareness  nowadays  is  widely 
regarded  as  a  valuable  tool  by  the  military  R&D 
community,  there  is  still  little  quantitative  evidence 
available  that  supports  the  acclaimed  benefits  of  generic 
fusion  methodologies  and  thus  justifies  their  application. 
It  is  therefore  legitimate  to  ask  how  much  value  sensor 
fusion  adds  to  the  observation  process.  In  case  studies 
this  question  is  seldom  addressed  and  it  is  therefore 
tempting  to  ask,  whether  the  acclaimed  advantages  of 
sensorfusion  are  “hype”  or  real.  In  this  paper  we  will  try 
to  identify  in  what  situation  the  application  of  sensor 
fusion  techniques  may  be  beneficial  and  which  sensor 
fusion  techniques  seem  to  be  among  the  most  promising. 
The  starting  point  of  our  discussion  is  the  observation 
that  the  added  value  of  fusion  of  similar  sensors  must 
originate  from  a  nonlinear  combination  of  sensor  data 
streams.  This  observation  naturally  gives  rise  to  the 
application  of  novel  nonlinear  models,  e.g.  from  the  area 
of  softcomputing,  viz-  fuzzy  logic,  neural  networks  and 
evolutionary  programming.  In  addition  a  trend  in 
research  can  be  observed  to  apply  fusion  techniques 
almost  exclusively  to  “high-level”  fusion  (i.e.  levels  2 
and  3  of  the  JDL  data  fusion  process  model),  implicitly 
considering  fusion  on  the  sensor  level  to  be  much  less  of 
a  challenge.  Motivated  by  the  fact  that  high  quality 
primary  data  are  of  paramount  importance  to  the  success 
of  all  subsequent  signal  processing  stages,  we  argue 
against  this  view  in  this  paper.  In  contrast  to  the 
classical  data  fusion  approach,  at  TNO-FEL  it  was 
decided  to  start  up  a  sensor  fusion  approach,  in  which  an 
integrated  approach  towards  sensor  optimization,  sensor 
management,  and  early  fusion  is  pursued.  Only  in  this 
way  one  may  hope  to  attain  the  goal  of  sensor  fusion, 
viz ■  an  improved  situation  assessment.  Concentrating  on 
the  primary  information  sources,  i.e.  the  sensors,  and 
using  their  synergy  at  the  earliest  stages  of  signal 
processing,  inevitably  forces  one  to  invest  in  state-of-the 
art  data-acquisition  and  real-time  (pre)processing 
facilities.  This  will  be  illustrated  by  discussing  some  of 


the  design  issues  of  the  FEL  sensor  fusion  demonstrator 
(SFD)  testbed.  The  SFD  project  has  been  started  up  by 
the  end  of  1996  and  initial  concepts  have  meanwhile 
been  defined.  The  hardware  and  software  concepts  of 
this  testbed  will  be  described  as  well  as  the  functionality 
that  we  hope  to  implement. 

2.  INTRODUCTION 

In  recent  years,  numerous  research  papers  have 
been  published  dealing  with  the  application  of 
multisensor  data  fusion,  also  referred  to  as 
distributed  sensing  high-level  fusion,  especially  in 
the  domain  of  military  observations  [1-6]. 
Although  intuitively  appealing,  one  may  conclude 
that  data  fusion  did  not  yet  bring  about  the 
expected  breakthrough.  Several  explanations  for 
this  can  be  given,  such  as  the  particularity  of  the 
application  domain,  the  limited  availability  of 
general  nonlinear  methods  for  fusion,  and  finally 
the  quality  of  the  primary  ‘raw’  sensor  data. 
Another  problem  may  be  the  unrealistic 
expectations  of  the  virtues  of  the  synergy  of 
multiple  sensors. 

The  absence  of  a  general  way  to  approach  the 
subject  and  the  many  ad  hoc  experiments  and 
simulations  that  have  been  published,  have 
motivated  the  present  work  at  TNO-FEL.  In  the 
following  we  will  shortly  review  the  history  of 
fusion,  define  sensor  fusion  as  a  field  of  research 
in  its  own  right,  and  in  Sec.  3  motivate  the  need 
for  a  sensor  fusion  demonstrator  (SFD)  testbed  at 
TNO-FEL,  sketch  the  problem  of  how  to  model 
sensor  fusion  and  suggest  some  directions  for 
answering  some  of  the  pertinent  questions  in  this 
field  using  concepts  from  soft  computing. 
Especially  the  use  of  fuzzy  measures  looks 
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promising  as  a  way  to  model  the  sensor  fusion 
process  quantitatively.  Next  in  Sec.  4  we  describe 
some  of  the  implementation  issues  and  the  plans 
for  the  first  test  experiment. 

2.1  Historical  overview 

Historically  the  idea  of  sensor  fusion  is  not  new: 
As  early  as  the  sixties  multi-radar  trackers  have 
been  in  use  by  the  military  for  air  traffic  control 
and  air  defense.  Multisensor  data  fusion  seeks  to 
combine  information  generated  by  multiple 
sensors  to  achieve  goals  that  would  be  very  hard  or 
impossible  to  achieve  with  single  sensors.  From 
the  point  of  view  of  efficiency,  scheduling, 
accuracy,  and  redundancy  it  seems  intuitively 
obvious  that  several  sensors  are  ‘better’  than  a 
single  sensor. 

Nowadays  data  fusion  is  a  well-accepted  method 
for  making  superior  inferences  in  the  field  of 
industrial  automation  ( e.g .  for  controlling  a  power 
plant,  an  oil  refinery,  a  cement  kiln  (for  a  review 
on  industrial  applications,  see  e.g.  [7,8]),  or  even  a 
nuclear  reactor  [9,10],  and  for  carrying  out  real¬ 
time  pattern  recognition  in  industry  using  a  variety 
of  sensors.  Especially  since  the  advent  of 
softcomputing  methods,  such  as  fuzzy  logic,  data 
fusion  has  become  a  widely  accepted  successful 
fusion  technology  in  industry.  We  note  however 
that  the  success  of  such  methods  is  primarily  due 
to  their  ability  to  model  human  behavior  or 
expertise  in  supervisory  control.  Sensor  fusion  also 
endeavors  to  mimic  cognitive  processes  in  humans 
by  absorbing  the  signals  of  the  human  observation 
system,  our  five  senses,  from  the  real  world  and 
integrate,  or  ‘fuse’,  these  signal  streams  to  arrive 
at  a  coherent  picture  of  our  environment.  As  such, 
sensor  fusion  is  concerned  with  lower  abstraction 
levels,  much  higher  information  rates,  and 
generally  requiring  faster  response  than  the  data 
fusion  used  in  supervisory  control  systems.  This 
forms  also  the  key  problem  in  applying  soft 
computing  methods  to  this  field:  in  controlling 
complex  industrial  or  organizational  processes  at 
relatively  long  timescales  human  operators  have 
accumulated  over  the  years  ample  experience.  In 
contrast,  there  is  only  limited  insight  in  the  way  a 
human  being  builds  up  an  environmental  picture, 
his  awareness,  from  continuous  multisensate 
observations.  It  can  therefore  be  a  useful  approach 
when  setting  up  a  sensor  fusion  testbed  to  have  a 
close  look  at  how  the  human  cognitive  system 


works.  Cognition  is  still  far  from  understood  in 
detail,  but  a  few  global  characteristics  are 
apparent:  the  human  recognition  system  consists  of 
a  massively  parallel  processor  that  merges  vague, 
qualitative  inputs  and  a  priori  knowledge,  acquired 
by  learning  from  experience  into  a  more  or  less 
consistent  picture  of  the  environment.  It  consists  of 
a  large  number  of  hierarchically  ordered  decision 
processes  running  concurrently,  simultaneously 
inferencing  on  the  same  set  of  input  data  at 
different  levels  of  granularity.  We  will  not  discuss 
these  points  in  detail  here,  since  they  are  outside 
the  scope  of  this  article.  With  the  properties 
mentioned  above  in  mind  we  will  attempt  to 
emulate  in  the  sensor  fusion  testbed  some  of  the 
functionality  that  we  observe  in  human  perception. 

Although  sensor  fusion  is  important  to  virtually  all 
phenomenological  sciences  and  engineering 
disciplines,  most  work  until  now  has  been  done  in 
the  field  of  defense  research.  One  reason  for  this  can 
be  understood  as  follows.  In  analytical  approaches, 
e.g.  in  a  physics  experiment,  the  measured 
quantities  or  interactions  are  often  so  small  that  the 
experimental  setup  has  to  be  designed  in  such  a  way 
as  to  make  sure  that  the  desired  quantity  or  effect  is 
optimally  measurable.  If  the  measured  quantities  are 
small,  the  experiment  is  repeated  many  times  and 
ergodicity  (see  Sec.  3.4)  and  statistics  are  used  to 
arrive  at  average  values  with  low  relative  standard 
deviation.  Especially  in  case  one  tries  to  prove  or 
disprove  the  correctness  of  a  theoretical  model,  this 
often  is  a  good  approach.  A  final  point  to  note  here 
is  that  -  apart  from  intrinsic  physical  real-time 
aspects  -  such  experiments  very  often  can  be 
repeated  many  times  and  real  time  constraints  are 
not  a  bottleneck. 

In  engineering  approaches  the  use  of  sensors  is 
more  synthetic,  as  illustrated  e.g.  in  the  field  of 
factory  automation.  Here  one  deals  with  a  well- 
defined  problem  such  as  the  quality  control  of 
products  on  a  manufacturing  line,  e.g.  checking  the 
soldering  joints  on  a  PCB  with  an  automated  vision 
system.  This  problem  certainly  has  real-time 
aspects,  but  the  optimization  can  be  done  offline 
and  the  observation  circumstances,  like  in  the 
physics  experiment,  can  be  optimized  offline,  e.g. 
by  testing  the  best  combination  of  sensors,  the 
proper  cameras  and  illumination,  parallel  operation 
with  more  than  one  quality  control  station  if  the 
speed  of  production  requires  so. 


31-3 


Figure  1  A  taxonomy  of  the  different  types  of  uncertainty. 


In  military  observations  we  deal  with  a  situation  that 
is  far  less  comfortable  than  the  situations  described 
above:  generally  speaking  it  is  necessary  to  assess  in 
real  time  an  often  complex  situation,  that  almost 
certainly  is  outside  one’s  complete  control. 
Handling  such  observations  requires  the  modeling 
of  uncertainty.  Apart  from  the  ordinary  problems 
such  as  noise  and  clutter,  radar  and  electro-optical 
sensors  operate  also  under  adverse  weather  and 
atmospheric  conditions,  without  any  possibility  to 
improve  the  circumstances  of  the  experiment,  or  to 
repeat  the  experiment,  under  strict  real  time 
constraints,  with  sometimes  enormous 
consequences  of  false  classification  and  even  more 
serious  penalties  for  non-detection.  In  addition,  by 
the  nature  of  the  military  metier,  most  interesting 
targets  move  at  high  speeds,  try  to  avoid  actively 
or  passively  or  mislead  sensors  by  jamming  or 
using  decoys  and  they  are  designed  in  such  a  way 
as  to  present  a  minimal  scattering  cross  section  to 
commonly  used  sensors  and  thus  to  be  virtually 
invisible  (‘stealth’). 

Under  these  circumstances  it  is  clear  that  doing 
military  observations  invariably  implies  the 
modeling  of  uncertainty.  Classically  this  is  often 
done  by  applying  statistical  methods,  notably 
Bayes’  theorem  to  formulate  a  (multi-)  hypothesis 
testing  problem  [10].  It  is  however  clear  that 
statistical  uncertainty  can  only  model  part  of  the 
uncertainty.  The  different  types  of  uncertainty, 
whose  measures  are  now  well  established  in 
classical  set  theory,  fuzzy  set  theory,  probability 
theory,  possibility  theory  and  evidence  theory  [11] 
are  schematically  summarized  in  Fig.l  The 
breakdown  distinguishes  fuzziness,  or  vagueness 


due  to  a  lack  of  definite  and  sharp  conceptual 
distinctions  and  ambiguity ,  the  situation  where  we 
are  dealing  with  one-to-many  relationships  in  the 
information  obtained  from  sensors,  yielding  non- 
specifity  in  the  case  that  the  data  leaves  two  or 
more  alternatives  unspecified,  or  even  discord,  i.e. 
disagreement  in  choosing  from  among  several 
alternatives. 

Recently  methods  that  explicitly  deal  with 
ambiguity  and  partially  overlapping  hypotheses 
such  as  Dempster  Shafer  theory  [12,13]  and  the 
application  of  belief  functions  instead  of 
probability  densities  have  become  popular.  Of 
even  more  recent  date  is  the  application  of  general 
fuzzy  measures  [14].  The  difficulty  inherent  to 
making  accurate  observations  in  military 
applications  and  the  lack  of  measurement  statistics 
are  the  prime  motivations  to  improve  single  sensor 
observations  by  merging  (partial) 
inferences/conclusions  from  one  sensor  with 
inferences  from  the  another  one. 

In  addition  recent  history  shows  that  the  nature  of 
military  operations  changes  rapidly:  Although 
sensors  are  vital  to  the  success  of  any  military 
mission,  it  becomes  at  the  same  time  much  more 
difficult  to  interpret  these  observations.  This  can 
be  explained  by  the  introduction  of  stealth 
technologies  (radar),  by  which  planes  become 
much  more  difficult  to  detect  by  radar,  the 
subtleties  of  ‘peacekeeping’  missions  compared  to 
classical,  full  scale  warfare  scenarios,  and  finally 
the  complexities  and  greater  vulnerability  of  navy 
vessels  operating  close  to  shore  (‘littoral  warfare’). 
Finally  it  should  be  noted  that  there  is  a  genuine 
need  to  fuse  sensor  generated  information,  at  least 
at  the  higher  levels  of  command  and  control:  the 
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man-machine  interface  being  the  limiting  factor. 
Although  new  sensors  have  been  developed  (e.g. 
GPS)  and  accuracy  and  resolution  in  space  and 
time  of  most  existing  sensors  have  greatly 
increased  in  time,  the  bandwidth  of  the  man- 
machine  interface  has  not.  The  situation  of  having 
to  deal  with  more  information  than  one  can 
process  in  a  certain  time  is  not  unsimilar  to  the 
situation  where  a  lack  of  information  exists.  Both 
situations  involve  taking  decisions  in  the  presence 
of  uncertainty  and  would  benefit  from  intelligent 
data  reduction  techniques,  such  as  sensor  fusion. 

2.2  Definition  of  sensor  fusion  vs.  data 
fusion 

Following  the  definition  of  the  functional  model  of 
the  data  fusion  process  that  is  widely  accepted  in 
the  military  research  community,  as  e.g.  presented 
by  Hall  [15],  and  using  the  terminology  as  agreed 
by  Joint  Directors  of  Laboratories  (JDL)  of  the 
Data  Fusion  Subpanel.  Multisensor  data  fusion  is 
by  the  JDL  defined  as 

“A  continuous  process  of  dealing  with  association, 
correlation  and  combination  of  data  and 
information  from  multiple  sources  to  achieve 
refined  entity  position  and  identity  estimates,  and 
complete  and  timely  assessments  of  resulting 
situations  and  threats,  and  their  significance.  ” 


We  will  in  this  paper  define  multisensor  fusion  as 
level  1  processing,  with  basic  processes:  data 
alignment,  association  and  correlation,  positional 
and  identity  fusion,  complemented  by  the  real-time 
part  of  level  4  processing  (“maintenance”).  The 
reason  for  including  part  of  level  4  may  perhaps 
seem  strange  at  first  sight,  but  is  immediately 
apparent  when  maintenance  is  interpreted  as  the 
assessment  of  the  status  of  each  sensor  in  order  to 
keep  it  optimally  tuned.  Apart  from  optimizing 
individual  sensors  monitoring  sensor  performance 
makes  it  possible  to  perform  ‘ sensor  management’ , 
i.e.  to  optimize  groups  of  sensors,  which  is  e.g. 
important  for  military  observation  systems  where  a 
large  number  of  sensors  co-operate  in  a  coherent 
way  and  in  which  part  of  the  sensors  may  be 
damaged  during  operation  ( c.f  a  phased  array  radar 
system).  The  sensor  management  system  contributes 
therefore  directly  to  the  robustness  of  a  system. 

After  having  clarified  the  scope  of  sensor  fusion, 
we  will  next  discuss  the  motivation  to  develop  a 
sensor  fusion  testbed,  some  of  the  problems 
inherent  to  sensor  fusion  and  the  type  of  approach 
that  has  been  selected  to  study  the  added  value  of 
sensor  fusion.  In  Fig. 2  a  schematic  representation 
is  given  of  the  different  levels  of  fusion  that  can 
be  distinguished  in  data  fusion.  For  clarity  the 
feedback  paths  originating  from  maintenance  and 
on-line  sensor  optimization  (fast  level  4 


Figure  2  The  JDL  process  model  for  data  fusion.  The  darker  areas  indicate  the  scope  of  the 
SFD. 
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processing)  have  been  included.  Early  sensor 
fusion  can  be  viewed  as  a  two-step  process,  a 
direct  fusion  step  followed  by  a  complimentary 
fusion  step  (Fig.3).  The  direct  fusion  process  acts 
immediately  on  raw  sensor  data,  after  a  possible 
preprocessing  stage  for  alignment.  This  type  of 
fusion  is  in  practice  limited  to  combining  signals 
from  similar  sensors.  In  the  next  stage,  in  the 
complementary  fusion  process,  very  different 
types  of  sensors  can  be  fused.  In  this  stage  it  is 
possible  that  a  considerable  data  reduction  is 
achieved  and  that  the  information  can  be 
represented  as  a  vector  in  feature  space.  Features 
such  as  range,  position,  orientation,  effective  cross 
section,  shape,  color,  etc.  are  extracted  from  the 
different  sensors  and  combined  in  qualitative  or 
quantitative  ways.  Combining  of  information  from 
complimentary  sensors  can  thus  be  seen  as 
augmenting  the  dimensionality  of  the  feature 
space.  After  this  fusion  step  all  sensor  information 
has  been  fused  and  one  needs  now  to  combine 
feature  vectors  with  existing,  a  priori  information 
about  the  environment,  collected  from  previously 
measured  data  or  intelligence.  This  more  abstract 
fusion  stage  is  typical  for  levels  2  and  3  of  the  JDL 
model  and  will  not  be  considered  in  this  article. 


3  SENSOR  FUSION 

3.1  Motivation  for  designing  the  SFD 
testbed 

The  general  motivation  why  multisensor  fusion  is 
pursued  is  generally  answered  in  terms  of  “to 
improve  the  combined  observations  of  different 
sensors  and  thus  create  a  better  situational 
awareness”.  Basically  this  boils  down  to  a  purely 
economic  factor.  From  the  operational  point  of 
view  possible  benefits  include  a  greater  user 
friendliness  because  of  data  reduction,  a  greater 
robustness  of  the  resulting  observation  system, 
higher  reliability  by  a  higher  plot  rate  and 
therefore  a  better  performance  of  e.g.  a  tracking 
algorithm,  and  thus  a  better  observation,  also 
under  adverse  circumstances.  Scientifically 
however  one  immediately  faces  difficulties  with 
this  formulation  of  the  motivation,  because  it 
introduces  subjectivity:  How  can  one  measure 
“situational  awareness”?  An  even  more 
complicated  question  to  be  answered  is  what 
exactly  can  be  improved  in  conventional,  single 
sensor  observation  and  how  different  sensors  can 
benefit  from  each  others  measurements. 
Answering  these  two  questions  is  by  no  means 
trivial  and  they  can  in  fact  only  be  answered  by 
considering  a  specific  application.  However  before 


Fusion  of  similar  sensors  Fusion  of  different  sensors  Merging  a  priori  knowledge 


Figure  3  Global  information  flow  in  sensor  fusion;  the  level  of  abstraction  in  the  fusion  process 
increases  to  the  right;  to  the  left  the  data  rate  increases. 
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doing  so,  it's  worthwhile  to  take  a  moment  and 
review  the  various  abstraction  levels,  methods, 
hardware  and  software  implementation  methods  of 
observation  systems  and  the  different  ways  that 
they  can  interact  and  cooperate.  The  basic 
difficulty  is  in  the  opinion  of  the  author  the 
absence  of  an  unique  theoretical  framework  for 
objectively  combining  the  information  streams 
generated  by  the  various  sensors.  The  difficulty 
lies  essentially  in  adequately  describing  the 
information  content  in  each  sensor  stream.  The 
amount  of  useful  information  in  a  data  stream 
created  by  a  sensor  is  of  course  dependent  on  the 
ultimate  goal  of  the  complex  of  observations. 
Whenever  this  goal  cannot  be  formulated  in  a 
clear,  transparent  and  unambiguous  way,  it  will  be 
extremely  difficult  to  develop  synergy  between  the 
sensors  and  to  compare  the  performance  of  the 
various  fusion  algorithms.  We  may  therefore 
conclude  that  the  order  to  study  sensor  fusion  and 
designing  a  sensor  fusion  demonstrator  (SFD)  due 
attention  must  be  paid  to  the  following  points: 

•  selection  of  suitable  fusion  experiments 

•  the  goals  to  be  improved  by  fusion 

•  selection  of  the  quantities  to  be  observed 

•  selection  of  number  and  type  of  sensors 

•  optimization  of  the  individual  sensors 

•  the  handling  of  uncertainty 

•  architecture  of  the  multi-sensor  fusion  setup 

•  selection  of  pre-processing  algorithms 

•  clear  distinction  of  objective  and  subjective 
modeling  issues 

In  this  context  it  is  worth  noting  that  optimization 
of  individual  sensors  is  a  first  step  for 
understanding  the  benefits,  and  the  limitations  of 
sensor  fusion,  because  if  individual  sensor 
observations  have  a  marginal  accuracy  or  do  not 
pertain  to  the  fusion  goals,  it  is  not  realistic  to 
expect  improvements  by  combining  these  streams. 
If  on  the  other  hand  the  observations  of  the 
individual  sensors  are  excellent,  their  is  very  little 
room  for  improvement.  The  ideal  sensor  fusion 
experiment  shall  therefore  make  use  of  well- 
defined,  accurate  and  reproducible  sensors  that, 
when  taking  individual  measurements  are  by 
themselves  not  effective  to  achieve  the  fusion  goal. 
Note  that  this  formulation  also  includes  the 
situation  that  an  individual  sensor  may  ultimately 
attain  the  fusion  goal  by  itself,  but  that  the 
recognition  or  detection  process  can  be 
substantially  sped  up  by  fusion. 


3.2  Benefits  and  limitations  of  sensor  fusion. 

A  number  of  potential  benefits  of  fusion  have  been 
compiled  by  Waltz  [16]  and  include 

1 .  Robust  operational  performance 

2.  Extended  spatial  coverage 

3.  Extended  temporal  coverage 

4.  Increased  confidence 

5.  reduced  ambiguity 

6.  Improved  detection 

7.  Enhanced  spatial  resolution 

8.  Improved  system  reliability 

9.  Increased  dimensionality 

We  first  note  that  most  of  the  benefits  quoted  in 
literature  are  benefits  that  are  exclusively 
associated  with  the  presence  of  multiple  sensors; 
they  are  not  the  benefits  of  sensor  fusion.  Most  of 
these  benefits  are  qualitatively  and  intuitively 
immediately  clear.  Globally  we  can  distinguish 
three  types  of  benefits: 

•  In  the  first  place  the  effect  of  multiple  sensors 
is  an  extended  spatial,  temporal,  or  spectral 
coverage  of  the  associated  phenomenon 
(benefits  2,3). 

•  A  second  type  of  benefit  follows  from 
statistical  arguments:  multiple  sensors  increase 
the  measurement  accuracy  (7)  and  from  this  an 
increased  confidence  (4)  may  be  derived,  or  at 
least  a  reduction  in  the  number  of  hypotheses 
about  the  targets  (5)  and  thus  an  improved 
detection,  c.q.  a  shorter  detection  time  (6).  Only 
in  the  cases  5  and  6  a  sensor  fusion  step  is 
needed. 

•  Finally  multiple  sensors  create  overlap  in 
observations  and  thus  redundancy.  If  this 
redundancy  is  properly  exploited  in  the  system 
design,  the  maintenance  (level  4)  module  will 
optimize  the  sensor  scheduling  and  will  result 
in  the  graceful  degradation  of  system 
performance  if  sensors  breakdown  (1,8,9). 

A  quantitative  aim  of  sensor  fusion  is  to  improve 
the  accuracy  of  the  observation,  e.g.  the  position  of 
a  target.  In  Fig.  4  this  situation  is  illustrated  by 
combining  a  forward-looking  IR  sensor  (FLIR)  and 
a  radar.  The  position  determination  of  the  radar 
and  the  FLIR  observations  is  given  together  with 
the 
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Figure  4  Reduction  of  positional  uncertainty 
radar  and  FUR. 


result  of  sensor  fusion.  The  inaccuracy  in  azimuth 
and  elevation  of  the  radar  sensor  is  compensated 
by  the  more  accurate  measurements  of  the  FLIR 
sensor,  while  the  pulsed  radar  accurately 
determines  the  range.  This  example  illustrates  how 
a  radar  can  initially  detect  a  target,  because  its 
wider  field  of  view.  Subsequently  the  FLIR  can  be 
cued  using  the 

inaccurate  coordinates  of  the  radar  to  initiate  the 
FLIR  measurement.  Together  they  determine  a 
small  region  of  interest  around  the  target,  so  that 
the  benefit  of  fusion  is  an  improved  estimate  (or 
reduced  uncertainty)  of  the  position  of  the  target. 

An  interesting  attempt  to  illustrate  in  a  quantitative 
way  the  virtues  of  sensor  fusion  is  described  in 
[17].  In  the  article  an  odd  number  N  of  identical 
sensors  are  fused  with  the  aim  to  classify  an 
observed  phenomenon  following  a  majority  vote 
rule.  The  sensors  are  assumed  to  be  statistically 
independent  and  the  a  priori  probabilities  are  taken 
equal  to  1/N,  corresponding  to  the  principle  of 
maximum  entropy,  equivalent  with  a  minimum  of 
a  priori  knowledge.  Although  the  example  is  a 
very  idealized  model  case  of  identical, 


by  combining  two  complementary  sensors: 


independent,  unbiased  sensors,  all  following  the 
same  statistics,  and  using  binary  classification  and 
a  majority  vote  scheme  as  fusion  aggregator,  a 
number  of  qualitative  results  are  worth  mentioning 
here: 

•  Fusing  data  from  multiple  sensors  (each  with 
an  individual  probability  of  correct  detection 
c.q.  classification  of  less  than  0.5)  results  in  a 
decrease  in  performance  in  going  from  a 
single  sensor  classification  to  the  multiple 
sensor  fused  result. 

•  If  the  individual  sensors  are  very  accurate 
(probability  of  correct  detection  larger  than 
0.95)  sensor  fusion  cannot  significantly 
improve  the  results  of  the  inference  process. 

•  The  relative  improvement  in  performance  of 
an  N  sensor  fusion  process  over  single  sensor 
performance  increases  as  a  function  of  N 
leveling  off  at  about  N=10.  Adding  more, 
identical  sensors  does  not  pay  off  beyond  this 
number,  (see  Fig. 5) 

•  The  maximum  relative  improvement  of  N 
sensor  performance  for  N  — >  °°  compared  to  a 
single  sensor  is  of  the  order  of  15-25  %, 
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depending  on  the  fusion  scheme.  The 
maximum  is  assumed  if  the  single-sensor 
probability  of  correct  detection  is  in  the  range 
between  0.60  and  0.75. 

Of  course  the  numbers  mentioned  above  should  be 
treated  with  care  because  they  depend  on  the  type 
of  aggregation  operator  chosen  to  represent  the 
fusion  process.  Moreover  these  conditions  refer  to 
the  simplified  case  of  identical  sensors,  i.e.  same 
positioning,  calibration  statistics,  biases,  sampling 
rates,  bandwidths,  sensitivities,  dynamic  behavior 
and  the  same  measured  quantities.  If  a  new  type  of 
sensor  is  added  to  the  sensor  suite,  the 
dimensionality  of  the  observation  is  increased  and 
a  substantial  increase  in  information  content  may 
be  expected,  depending  on  fusion  objective. 
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Figure  5  Marginal  gain  in  the  probability  of 
correct  classification  by  adding  sensors  in 
going  from  1  to  3,  3  to  5,  and  5  to  7  sensors 
as  a  function  of  the  single  sensor  probability 
P,  as  calculated  in  [17]. 

In  another  recent  study  [18]  the  dependence  of 
decision  reliability  on  different  fusion  strategies 
for  the  case  of  2  different  sensors  (i.e.  sensors  that 
are  not  identical)  has  been  investigated  based  upon 
simple  Boolean-type  operators,  along  the  lines  of  a 
probabilistic  detection  system  (two  hypotheses  that 
are  mutually  exclusive  and  span  up  the  whole 
universe  of  discourse).  In  a  way  this  study  allows 
to  extend  the  validity  of  the  rules  formulated  above 
to  more  subtle  fusion  strategies  than  the  'majority 
vote'.  Also  from  these  results  we  may  conclude 


that  for  fusion  to  be  successful  the  single  sensor 
detection  probability  must  be  higher  than  0.50  and 
that,  in  the  case  of  two-sensor  fusion,  increases  in 
detection  performance  of  maximum  1 1  %  (from  a 
single-sensor  probability  of  detection  Pi  of  0.8  to  a 
two-sensor  fused  probability  of  detection  P2  of 
0.899),  or  17%  (an  increase  from  Pi  =  0.80  to  P2  = 
0.941)  have  been  calculated,  depending  on  the 
fusion  method. 

Although  these  results  do  in  no  way  preclude  a 
substantial  increase  in  the  combined  sensor 
performance  brought  about  by  a  suitably  chosen 
sensor  fusion  strategy,  these  numbers  indicate  that 
if  measurements  from  identical,  hard  decision 
making  sensors  are  combined  under  the 
assumption  of  statistical  independence,  the 
marginal  gain  in  performance  will  be  limited  to  a 
percentage  of  a  few  tens,  say  10-20%  . 


3.3  The  sensor  fusion  problem 

The  process  of  sensor  fusion  can  be  viewed  in  a 
number  of  different  ways,  depending  on  the  goal 
that  one  wants  to  achieve:  One  may  view  the 
synergy  between  sensors  as  a  scheduling  problem: 
How  can  one  schedule  the  available  resources  (an 
ensemble  of  sensors)  in  such  a  way  that  the 
observation  process  is  optimal  ( e.g .  that  a  typical 
category  of  targets  can  be  detected)?  This  clearly 
depends  on  the  availability  and  performance  of  the 
sensors  as  given  by  the  physical  properties  and 
capabilities  of  each  sensor,  but  in  a  military 
situation  the  availability  may  also  depends  on  such 
factors  as  physical  damage  or  jamming  of  sensors 
and  environmental  conditions  (e.g.  fog,  sunset, 
night,  and  “clutter”,  i.e.  (back) scattering  from 
land,  see  and  atmosphere).  Basically  the 
application  of  sensor  fusion  therefore  involves  the 
solution  of  an  multivariate,  generally  nonlinear, 
time-dependent  optimization  problem. 

It  is  worth  noting  in  this  context  is  the  importance  of 
optimization  as  a  keyword  in  sensor  fusion  at 
various  levels.  Above  we  have  already  mentioned 
‘optimal’  in  the  definition  of  fusion,  as  well  as  in 
the  remark  on  sensor  management.  Combining  these 
two  observations  naturally  gives  rise  to  the  study  of 
nonlinear  optimization,  a  field  that  has  since  long 
attracted  the  attention  of  researchers  from  very 
different  fields,  e.g.  mathematics,  game  theory,  AI, 
systems  theory,  information  science,  physics, 
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biology,  operations  research  and  various  other  areas, 
and  that  despite  all  these  efforts  still  holds  many 
secrets.  The  general  approach  in  arriving  at 
(approximate)  solutions  is  by  heuristic  methods. 
Exhaustive  search  is  generally  precluded  because 
even  small  practical  problems  already  suffer  from 
the  combinatorial  explosion. 

One  of  the  problems  involves  the  formulation  of 
the  function  to  be  optimized,  because  of  the 
difficulties  to  define  a  suitable  measure  for  such 
notions  as:  better  usage  of  resources  (economy), 
robustness  vs.  redundancy,  increasing  the 
statistical  accuracy  of  an  observation,  earlier, 
timelier  achievement  of  an  acceptable  level  of 
observational  accuracy.  For  all  these  notions  it  is 
possible  to  come  up  with  ad  hoc  definitions,  but 
often  it  is  felt  that  including  too  much  heuristics 
makes  the  solution  too  subjective.  A  final  point  to 
note  is  that  it  is  often  desirable  to  have  the 
possibility  to  delay  decisions  (classifications)  a 
while,  thereby  increasing  the  confidence  level  of 
the  final  decision  . 

The  nonlinear  combination  of  sensor  information 
holds  possibly  a  key  to  gaining  insights  in  the 
exploitation  of  sensor  fusion.  Nonlinearity  can  be 
introduced  in  signal  analysis  in  a  number  of  ways: 

•  Decision  making  is  a  nonlinear  process:  hard 
decision  making  consists  of  making  a  choice 
out  of  a  (finite)  set  of  alternatives,  in  its 
simplest  form  choosing  between  two  or  more 
mutual  exclusive  hypotheses  H  of  the  form:  “IF 
(signal  >  threshold)  THEN  (H  is  true)  ELSE 
(— iH  is  true)”,  where  H  u  — iH  =  U,  using 
binary  logic.  The  detection  of  a  target  basically 
follows  this  simple  scheme.  Statistics  is 
subsequently  applied  in  order  to  take  into 
account  the  stochastic  nature  of  sensor 
performance,  fluctuations  in  the  environment 
and  in  the  target  parameters,  and  notions  as 
“false  alarm  rate”,  “detection  probability”, 
“misclassification  rate”  are  introduced.  So  it 
appears  relatively  straightforward  to  view  the 
fusion  process  as  a  synergy  between  taking 
decisions  based  upon  two  or  more  sensors,  the 
difficulty  associated  with  crisp,  binary  type 
decisions  is  that  (too)  much  of  the  intrinsic 
sensor  information  is  discarded,  thus  making  it 
virtually  impossible  to  use  the  decision  in  a 
subsequent  decision  stage.  In  analogy  with  the 
paradigm  of  human  decision  making  one  needs 


to  introduce  two  additional  properties,  not 
present  in  the  classical  approach: 

1.  A  complex  decision  consists  of  a 
hierarchy  of  simpler,  partial  decisions. 

2.  In  order  to  create  a  continuous  mapping 
between  inputs  and  the  final  outcome, 
it  is  desirable  that  decisions  are  soft 
decisions,  e.g.  represented  by  a  fuzzy 
number  of  by  a  degree  of  membership 
associated  with  a  certain  hypothesis 
(see  e.g.  [19]).  In  this  way  a  smooth 
propagation  from  sensor  input  to  final 
decision  is  warranted.  The  use  of  soft 
decisions  has  the  added  advantage  of 
increased  robustness  and  ability  to 
handle  conflicting  information. 

•  An  alternative  to  soft  decision  taking  for 
constructing  an  accurate,  fused  decision  is  by 
delaying  the  final  decision  in  time  and  by 
recursively  reconsidering  the  decision  in  those 
cases  where  sensor  inputs  would  yield 
ambiguous  or,  at  least  partially,  conflicting 
decisions.  In  its  simplest  form  this  means  that  a 
three-valued  logic  has  to  be  selected  in  the  case 
of  having  to  make  a  binary  decision  ( i.e .  having 
to  choose  among  two  mutually  exclusive 
hypotheses  H,,  and  H,  that  cover  together  all 
cases:  H„n  H,  =  0  ).  Thus  a  third  hypothesis 
H2,  corresponding  to  “cannot  decide  yet 
between  the  alternatives”,  is  introduced.  If  the 
initial  decision  is  H2,  sensor  inputs  are  scanned 
again  in  the  next  data  acquisition  cycle  to 
resolve  the  outstanding  decision.  In  this  way 
one  may  construct  a  successive  refinement  of 
the  decision  by  which  a  better  support  for  the 
final  decision  is  created  at  the  cost  of  response 
time.  In  this  way  has  shown  that  it  is  possible  to 
obtain  positive  synergy  for  suitable  parameter 
regimes  and  after  enough  recursions  [18]. 

•  Once  we  allow  soft  decisions  to  exist  for  some 
time,  we  have  to  answer  the  question  what  type 
of  ‘fusion  operator’,  or  “fusor”,  should  be  used 
when  aggregating  soft,  partial  decisions,  and  if 
possible  select  the  ‘best’  aggregation  scheme. 
One  can  easily  imagine  that  this  is  not  an  easy 
question,  since  a  multitude  of  schemes  are 
possible  and  the  answer  is  clearly  context 
dependent.  In  [18]  several  schemes  have  been 
investigated.  Although  the  detailed  final 
decision  is  dependent  on  the  actual  fusion 
operator,  some  general  features  prevail.  All 
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fusors  have  in  common  that  they  are  nonlinear, 
ranging  from  fuzzy  generalizations  of  the 
Boolean  “AND”  and  “OR”  operator,  to  a 
“majority  decision”  fusion  scheme. 

These  observations  guide  us  in  the  search  for  an 
effective  sensor  fusion  algorithm.  The  relevance  of 
fuzzy  logic,  fuzzy  measures  and  fuzzy  aggregation 
operators  for  the  development  of  fundamental 
sensor  fusion  concepts  in  detection  and 
classification  is  striking,  the  optimum  way  how  to 
combine  information  from  different  sensors 
depends  on  the  particular  gals  of  the  mission  for 
which  the  observation  is  made,  as  well  as  on  the 
environmental  circumstances  and  the  properties, 
observables  and  relative  level  of  confidence  of  the 
individual  sensors.  This  will  be  the  subject  of 
studies  with  the  SFD.  Note  that  fuzzy  logic  is  not 
introduced  here  to  allow  for  human  vagueness  and 
subjectivity,  but  rather  to  get  rid  of  the  limitations 
of  Boolean  logic  operations  and  the  inherent  crisp 
decisions  at  early  stages  of  a  complex  decision 
tree. 


3.4  The  key  concept  of  sensor  fusion: 
Synergy 

A  general  concept  that  is  intimately  connected  to 
the  idea  of  fusion  is  that  of  ergodicity,  i.e.  the 
concept  that  the  outcome  of  an  observation  is 
unique,  independent  of  the  fact  that  one  makes  a 
series  of  consecutive  measurements  with  one 
system,  or  that  one  makes  N  parallel  setups  and 
combines  the  N  different  outputs  at  one  time.  In 
the  macroscopic  physical  world  the  concept  of 
ergodicity  generally  is  assumed  to  hold  without 
exceptions,  although  in  microscopic  physics  some 
examples  have  been  found  in  spinglasses,  where 
ergodicity  does  not  hold.  For  a  review  the  reader  is 
referred  to  [20],  Throughout  this  paper  we  will 
assume  ergodicity.  However  it  should  be  pointed 
out  that  this  statistical  principle  is  sometimes 
difficult  to  apply  in  the  real  world,  such  as  in 
military  observations,  because  of  the  fact  that  we 
are  mostly  dealing  with  single,  isolated  events. 

The  fusion  of  sensor  observations,  i.e.  the 
combination  of  observations  obtained  from  the 
same  sensor  at  different  times  (temporal  fusion),  or 
the  combination  of  simultaneous  observations  take 
by  a  number  of  equivalent  sensors  (repetition)  or 
the  combination  of  sensor  observations  with  a 


priori  information  obtained  from  previous 
measured  data,  is  the  focus  of  attention  in  the 
present  work.  The  key  concept  in  fusion  is  how  to 
take  advantage  of  nonlinearity. 

We  are  not  so  much  concerned  with  increasing  the 
accuracy  of  an  observation  by  repeating  a 
measurement  a  number  of  times  and  thus  reduce 
the  statistical  variation  of  the  average.  Rather  it  is 
our  objective  to  extract  additional  information  out 
of  this  data  set  (reduction  of  information)  by 
correlating  (  not  superimposing)  the  measurement 
with  observations  from  other  sources.  Moreover 
in  the  case  of  fast  moving  targets  it  is  generally 
impossible  to  obtain  a  sufficient  number  of 
samples  to  apply  statistics. 

Two  different  types  of  measurements  are  of 
interest  in  military  observation  systems: 

1.  determination  of  presence,  position, 
orientation,  and  speed  of  a  target  and 

2.  identification  and  recognition  of  a  target  ( type, 
affiliation  etc.) 

Although  identification  clearly  is  an  entirely 
different  characteristic  of  a  target  compared  to  its 
position  and  speed  and  although  the  latter  can 
generally  be  determined  at  much  larger  distances 
than  those  at  which  identification  can  the 
accomplished  with  reasonable  confidence, 
identification  can  help  improve  the  accuracy  with 
which  speed  can  be  determined  and  vice  versa.  In 
particular  the  identification  of  a  target  may  be 
helped  through  a  wealth  of  observations,  whereas 
establishing  the  position  and  speed  of  a  target  can 
only  be  accomplished  by  the  few  sensors.  The 
identification  of  a  target  will  be  accomplished 
more  easily,  because  of  the  higher  dimensionality 
of  the  ‘feature  vector’,  provided  that  a  good 
database  of  properties  is  available.  The  basic 
problem  in  recognition  is  to  exploit  the  high 
dimensionality  and  representing  data  in  such  a  way 
that  differentiation  between  various  possibilities 
becomes  easier.  Therefore  the  task  of  sensor 
fusion  is  the  combination  of,  possibly 
incompatible,  measurements  and  to  try  and 
construct  from  these  an  improved  environmental 
picture.  In  this  modeling  one  needs  to  include  the 
confidence  level  of  the  new  measurement,  as  well 
as  how  to  combine  this  information  with  the 
already  existing  picture.  Various  schemes  have  be 
proposed  in  the  past:  e.g.  Bayes’  rule  of 
combination  from  statistics,  belief  measures,  and 


31-11 


Dempster  Shafer  theory.  Although  most  of  these 
methods  have  a  sound  theoretical  basis,  their 
application  sometimes  lacks  theoretical 
justification  or  simply  yield  non-intuitive  results 
in  specific  situations.  This  makes  it  difficult  to 
compare  results  that  are  obtained  with  different 
methods. 

3.5  A  systematic  approach  to  Level  1 
processing 

In  this  section  we  will  outline  a  practical  approach 
to  sensor  fusion  in  military  observation,  following 
the  theoretical  framework  referred  to  as  Level  1 


data  fusion  in  the  JDL  model.  Two  generic  tasks 
are  of  importance  in  almost  all  observation 
processes,  viz.  detection  and  recognition.  Despite 
the  fact  that  different  sensors  and  methods 
commonly  used  to  accomplish  these  tasks,  it  is 
obvious  that  the  successful  completion  of  one  task 
will  almost  certainly  have  a  positive  effect  on  the 
other  one.  If  however  the  identification  and 
recognition  tasks  are  considered  to  be  “hard” 
decisions  that  result  from  the  independent 
processing  of  separate  sensors,  interaction  of  the 
two  processes  can  only  take  place  after  the  first 
process  has  reached  a  decision.  In  executing  more 
complex  tasks  we  have  already  indicated  in  Sec 
3.3  that  it  may  be  more  advantageous  to  allow  for 
partial,  delayed,  or  “soft”  decisions  which  may 


Table  1.  Typical  operational  characteristics  of  military  observation  sensors. 


SENSOR  CHARACTERISTIC 

DESCRIPTION 

Detection  performance 

Detection  characteristics,  e.g.  false  alarm  rate, 
detection  probability,  detection  range)  for  a 
calibrated  target  in  a  given  noise  background. 

Spatial  and  temporal  resolution 

Ability  to  distinguish  between  two  targets  that 
are  moving  close  together 

Spatial  coverage 

Spatial  volume  covered  by  a  sensor.  For  staring 
sensors  this  would  be  the  field  of  view,  for 
scanning  sensors  also  the  scan  pattern  must  be 
taken  into  account 

Detection/Tracking  modes 

Search  and  tracking  modes  performed: 

1 .  Staring  and  scanning 

2.  Single  or  multiple  target  tracking 

3.  Single  or  multimode  (track- while  scan/stare) 

Target  revisit  rates 

Rate  at  which  a  given  target  is  revisited  in  case 
of  a  scanning  sensor 

Measurement  accuracy 

Statistical  accuracy  of  sensor  measurements 

Measurement  dimensionality 

Number  of  independent  physical  observables 
that  can  be  measured  with  the  sensor 

Hard  vs.  soft  reporting 

Sensor  outputs  can  be  Boolean  (signal  above  or 
below  a  threshold)  in  the  case  of  hard  reporting, 
or  alternatively  they  can  make  a  soft  decision, 
i.e.  output  quantitatively  a  measure  to  what 
extent  a  hypothesis  is  supported  by  the 
measurement. 

Detection/  track  reporting 

Sensor  reports  each  individual  target  detection 
or  maintains  a  time  sequence  representation 
(track)  of  the  target's  trajectory. 

31-12 


offer  a  way  to  separate  the  various  goals  and  thus 
allow  us  to  break  down  a  complex  task  into  a 
hierarchy  of  relatively  simple  decisions.  Partial  or 
“soft”  decisions  can  be  combined  at  earlier 
moments  in  the  processing  chain  without 
discarding  too  much  information  and  thus  offer  a 
method  for  applying  early  fusion  of  sensor 
streams.  Before  discussing  in  more  detail  how  soft 
computing  methods  can  be  used  to  achieve  sensor 
fusion,  we  will  first  review  the  signal  processing 
steps  that  are  necessary  to  benefit  from  sensor 
fusion.  An  overview  of  the  physical  characteristics 
of  observation  sensors  that  are  relevant  for  defense 
applications  is  given  in  Table  1  (from  Hall  [15]). 
In  addition  sensors  can  be  free  running, 
synchronized  if  they  have  commensurate  sample 
rates,  or  can  be  operated  in  a  ‘master-slave’  mode. 

From  Table  1  it  can  be  seen  that  most  sensors  are 
decision-oriented  towards  achieving  the  detection, 
positioning  and  tracking  or  recognition  of  a  target. 
These  tasks  are  generally  considered  sequentially 
and  independently.  The  classical  way  in  which 
fusion  is  applied  is  by  transforming  a  physical 
measurement  into  some  hard  decision  ( e.g .  a  ‘plot’, 
‘track’,  ‘identity’,  etc.)  that  is  communicated  to  the 
user  via  the  man-machine  interface,  generally  an 
optical  display.  The  fusion  process  of  the 
information  shown  on  a  number  of  different 
displays  then  takes  place  in  the  mind  of  the 
operator,  who  assesses  the  situation,  or  makes  a 
threat  analysis.  All  these  fusion  processes  take 
place  in  the  human  mind,  after  the  sensor  signals 
have  been  processed  completely  (Fig  6a). 

A  first  step  towards  true  multi-sensor  fusion  is 
'late'  fusion  (Fig.  6b):  the  construction  of  a  special, 
goal-oriented  architecture  that  fuses  on  the  level  of 
images,  with  the  goal  to  enhance  the  image  {e.g. 
combining  IR  and  visible  light  images  using  some 
false  color  scheme),  or  to  fuse  the  consecutive 
plots  of  moving  targets  into  a  single  track  by 
taking  into  account  some  type  of  assumed  target 
dynamics,  or  the  fusion  of  tracks  generated  by 
different  sensors  {e.g.  two  radars  or  a  radar  and  an 
electro-optical  sensor).  The  results  are 
qualitatively  better,  especially  if  the  sensors  are 
one  different  platforms,  or,  alternatively,  under 
difficult  circumstances  (weather,  jamming)  when 
one  sensor  can  be  supplemented  by  the  other.  In 
the  case  of  early  fusion  with  many  same  sensors, 
one  can  process  directly  on  the  raw  data  without 
much  preprocessing,  possibly  except  for  bias 


removal  (Sec.  3.5.1).  In  this  sense  one  may  view 
SAR  (Synthetic  aperture  radar)  also  as  a  'temporal' 
fusion  process,  combining  different  samples  of 
single  sensor  to  simulate  a  kind  of  multi-sensor 
phased  array  with  a  much  larger  aperture  than  that 
of  the  actual  sensor. 

The  early  fusion  process  that  we  would  like  to 
investigate  with  the  SFD  is  schematically  given  in 
Fig.  6d:  a  collection  of  interacting  sensors,  each 
contributing  to  one  or  more  decision  trees  of 
positional  c.q.  feature  declarations.  Each  tree 
represents  a  hierarchical  decision  process,  building 
up  from  fast,  low  level,  noisy  decisions,  up  to 
well-founded,  abstract  decisions  that  require  some 
time  to  take.  The  sensor  fusion  architecture 
outlined  in  Fig.  6e  illustrates  the  ultimate  goal  of 
sensor  fusion:  mixed  early  and  late  fusion 
processes  on  different  levels  of  abstraction,  each 
subgoal  benefiting  from  its  own  support  set  of 
sensors  and  the  results  presented  on  one  display  to 
the  human  observer  in  a  representation  that  is 
highly  informative,  indicating  at  the  same  time 
alternatives,  as  well  as  the  associated  confidence 
levels. 

We  have  already  noted  in  Sec.  3.2  that  application 
of  multiple  sensors  in  general  can  be  beneficial  to 
the  accuracy  of  an  observation,  even  though  one 
low  resolution  sensor  directs  a  sensor  with  a 
higher  resolution  (“cueing”)  without  actually 
sensor  fusion  occurring.  For  making  hard  detection 
statements  explicit  knowledge  on  the  clutter  is 
required,  and  in  order  to  improve  single-sensor 
detection  capabilities,  one  needs  to  carefully 
analyze  the  model  assumptions  that  have  gone  into 
the  sensor  design.  New,  more  detailed  clutter 
models  corresponding  to  the  state  of  the  art  of 
sensor  technology,  may  be  needed  for  the 
detection  of  harder  targets.  On  top  of  that  sensor 
fusion  may  be  applied.  But  it  should  be  noted  that 
sensor  fusion  can  never  make  up  for  poor  clutter 
modeling. 

For  any  sensor  fusion  process  to  be  successful,  one 
has  to  properly  prepare  the  raw  signals  originating 
from  the  single  sensors.  From  a  system  point  of 
view  one  has  to  determine  the  stage  at  which 
fusion  has  to  take  place  (ranging  from  ‘early’  to 
‘late’)  in  relation  to  the  goal  that  must  be  achieved 
by  the  fusion  process.  Once  this  has  been  decided 
the  first  step  in  preparing  the  sensor  signals  for 
fusion  is  alignment,  to  guarantee  that  the  fields  of 
view  (FOV)  have  maximum  overlap.  Early  fusion 


Fusion  of  information  in  human  mind 


Late  fusion:  e.g.  Image  fusion,  T rack  fusion 


Early  fusion  with  similar  sensors,  e.g.  SAR 


Early  fusion  with  different  sensors  and  with  different  fusion  objectives 


Distributed  multisensor  fusion:  concurrent  early  and  late  fusion  on  different  levels 

Figure  6  Different  fusion  architectures  starting  from  late  fusion  (i.e.  in  the  human  mind), 
via  early,  multisensor  fusion  with  the  same  and  with  different  sensors  to  the  general  case 
of  multisensor  multiobjective  fusion  on  different  levels  concurrently. 
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can  only  be  successful  if  there  is  an  overlap 
between  the  FOVs.  The  second  step  is  the  proper 
correlation  of  the  same  objects  in  the  FOV  of  the 
sensors.  This  task  can  in  practice  be  quite 
laborious  when  many  targets  are  observed 
simultaneously  (large  FOV;  scanning  sensors). 
Before  any  of  these  preprocessing  transforms  and 
associations  can  be  carried  out  however,  attention 
has  to  be  paid  to  the  optimization  of  performance 
of  the  isolated  sensor  by  removal  of  the  biases  in 
the  observed  quantities. 

3.5.1  Bias  removal 

An  aspect  of  particular  importance  to  sensor  fusion 
is  that  of  on-line  bias  removal  [21].  As  discussed 
in  the  previous  sections,  sensor  fusion  is  expected 
to  improve  the  observation  process,  by  extracting 
more  information  from  the  sensors  than  could 
otherwise  be  done  by  processing  the  different 
sensor  streams  independently  and  therefore  to  be 
beneficial  by  enhancing  the  performance  of 
observation  and  detection  tasks,  especially  under 
difficult  circumstances.  However,  it  should  be 
noted  that  combining  information  from  different 
sensors  at  an  early  stage  initially  complicates  the 
analysis  compared  to  e.g.  combining  subsequent 
images  from  the  same  sensor,  or  to  the  separate 
processing  of  the  single  sensor  streams  and 
combining  the  results  at  a  later  stage,  i.e.  at  a  more 
abstract  level.  The  reason  for  this  is  that  the 
relative  importance  of  systematic  measurement 
errors  compared  to  accidental  errors  grows  when 
increasing  the  number  of  sensors.  Accidental 
measurement  errors  follow  some  type  of  stochastic 
and  will  average  out  to  zero  by  repeating  the 
measurement.  In  contrast,  systematic  errors  will 
persist  and  produce  a  bias  in  the  measured  quantity 
that  can  only  be  removed  by  careful  theoretical 
analysis  and  proper  design  of  the  experimental 
setup.  In  a  single  sensor  observation,  e.g.  a  video 
sequence  from  a  CCD  visible  light  camera,  bias 
errors  in  position  are  not  important  as  long  as  one 
is  only  interested  in  recognition  or  in  the  relative 
positions  of  objects  in  the  scene.  If  the  absolute 
position  of  a  target  should  be  estimated  from  the 
scene,  the  correct  estimation  of  bias  errors 
becomes  essential  and  many  more  data,  such  as  the 
positioning  of  the  tracking  platform  of  the  camera, 
the  specification  of  the  optics  and  the  observed 
spectral  range  become  important.  Typical 
quantities  that  are  sensitive  to  bias  errors  are 


position  coordinates.  As  soon  as  the  observations 
of  more  than  one  sensor  have  to  be  combined,  we 
are  confronted  with  differences  in  bias.  These  bias 
errors  originate  from  the  relative  positions  of  the 
sensors,  their  orientations,  different  fields  of  view, 
different  temporal  and  spatial  resolutions,  as  well 
as  from  differences  due  to  the  spectral  properties 
of  the  medium.  In  fusing  long  range  optical  and 
radar  observations  one  has  to  take  into  account 
that,  depending  on  the  specific  circumstances,  the 
optical  path  and  the  rf  path  to  the  same  target  are 
not  necessarily  straight  nor  equal  and  have  to  be 
accounted  for  ('alignment').  The  number  of  bias 
parameters  to  be  estimated  for  a  typical  sensor, 
such  as  a  radar  antenna,  could  be  as  high  as  ten 
[21],  This  number  illustrates  that  the  task  of 
estimating  the  various  systematic  errors  is  by  no 
means  a  trivial  one.  All  these  effects  have  to  be 
corrected  for  before  sensor  fusion  can  be  applied. 
If  the  preparatory  corrections  are  not  precise 
enough  they  could  easily  deteriorate  the 
performance  of  the  sensor  fusion  process,  or  even 
destroy  it  completely.  In  a  practical  situation  the 
removal  of  the  many  systematic  errors  contributing 
to  the  outputs  of  the  sensors  is  a  matter  of 
compromise:  because  the  high  data  rates  at  the 
sensor  level,  it  will  be  impossible  to  let  a  bias 
removal  process  run  at  the  same  pace  in  parallel 
with  the  data  acquisition  process.  Some  time 
averaging  will  be  necessary,  filtering  out  the 
highest  order  fluctuations.  On  the  other  hand,  if 
the  estimates  for  the  bias  levels  are  incorrect  or  too 
scarce,  the  effectiveness  of  the  sensor  fusion 
process  is  negatively  affected,  sometimes  even 
reducing  functionality  below  the  performance  of  a 
single  sensor  system.  An  example  of  this  is  the 
tracker  function  of  a  radar  sensor:  the  existence  of 
a  bias  error  in  the  position  vector  is  of  only  minor 
importance  in  a  single  tracker  system,  but  may 
result  in  creating  multiple  tracks  that  are 
originating  from  the  same  target  in  a  multi-radar 
tracking  system,  if  no  compensation  is  provided 
for  the  sensor  biases.  The  possible  benefits  of 
sensor  fusion,  e.g.  constructing  one  track  of  high 
accuracy  using  twice  the  number  of  plots,  are  thus 
precluded  in  the  preprocessing  stage,  even  before  a 
sensor  fusion  algorithm  could  be  applied. 
Therefore  raw  single  sensor  data  should  be 
compensated  for  bias  errors,  before  they  are 
associated  with  tracks.  In  [21]  a  system  has  been 
described  that  constructs  at  regular  intervals  a  set 
of  linear  equations  describing  to  first  order  how 
the  various  bias  errors  in  position  range  and 
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direction  of  the  sensor  affect  the  output  signal.  At 
regular  intervals  the  set  of  linear  equations  is 
solved  to  obtain  the  best  estimate  in  a  least  squares 
sense  of  the  single  sensor  bias  signals,  which  are 
then  used  to  filter  the  new  samples  obtained  in  the 
next  data  acquisition  cycle  before  applying  to  the 
sensor  fusion  process.  The  bias  update  rate  is 
much  slower  than  the  sampling  rate,  partly  because 
of  the  real-time  processing  constraints,  partly 
because  adopting  a  coarser  granularity  in  time 
effectively  limits  the  bandwidth  of  bias 
fluctuations.  This  is  motivated  by  the  assumption 
in  the  analysis  that  the  bias  is  constant  and 
therefore  its  fluctuations  are  by  definition  slow  on 
the  time  scale  of  the  measurement  process, 
including  the  sensor  fusion  processing.  This 
batchlike  bias  calculation  is  called  'quasi-recursive 
since  it  combines  the  structure  of  batch  processing 
with  an  infinite  batch  size,  because  of  the 
recursion. 


3.5.2  Fusion 

As  we  have  seen  in  the  previous  section,  it  is 
worth  to  select  the  proper  sensors  and  it  is  also 
necessary  to  spend  sufficient  effort  in  the 
preparation  of  the  signals  before  they  can  be  fused. 
It  should  be  noted  that  fusion  is  not  a  magic  way  to 
reduce  the  quality  or  the  price  of  a  sensor  and  still 
get  the  same  observation  performance.  Making 
accurate  observations  requires  a  great  deal  of 
study,  modeling  and  experimentation;  the 
successful  application  of  sensor  fusion  initially 
means  more  work  than  the  application  of  isolated 
sensors. 

Depending  on  what  the  goal  of  the  fusion  process 
is,  there  are  different  time  scales  to  consider: 

•  The  maximum  sampling  rate,  limiting  the 
highest  instantaneous  bandwidth  of  a  signal, 
which  is  essentially  a  measure  for  the  sensor 
resolution:  in  a  pulsed  radar  it  is  the  range 
resolution  and  in  a  camera  system  it  is  the 
transverse  2D  spatial  resolution. 

•  The  frame  rate.  This  rate  is  important  in 
extracting  information  from  a  time  sequence  of 
camera  images,  e.g.  with  optic  flow  analysis. 
By  using  the  temporal  correlation  of  objects  in 
the  pictures  when  the  platform  is  moving,  it  is 
possible  to  make  estimates  of  the  distance  of 
each  of  these  objects. 


•  The  batch  processing  rate:  If  signal  processing 
is  not  performed  on  a  continuous  basis,  as  is 
e.g.  the  case  of  Kalman  filtering,  one  is  not 
forced  to  make  a  priori  assumptions  and  may 
therefore  be  more  accurate  than  Kalman 
filtering  especially  in  early  stages  of  the  signal 
processing  chain.  A  disadvantage  is  that  batch 
processing  is  considerable  costlier  than  Kalman 
filtering  in  terms  of  processor  (CPU)  time. 

Although  we  will  here  focus  only  on  sensor  fusion, 
many  more  aspects  in  modeling  need  to  be 
considered,  before  an  attempt  to  apply  sensor 
fusion  should  be  made.  We  mention  here  only  a 
few: 

•  clutter  modeling:  the  type  of  statistics, 
correlation  times  and  correlation  lengths. 

•  construction  of  an  ‘a  priori’  environmental  data 
base,  necessary  to  make  (partial)  decisions  on 
identity  and  position. 

•  the  modeling  of  the  target  dynamics  if  the 
target  is  moving,  and  its  significance  for 
improving  classification. 

We  have  already  seen  in  Sec.  3.2  that  fusion  of 
multiple  same  sensors  mainly  improves  the 
statistics  of  the  observed  phenomenon  compared 
to  the  single  sensor  statistics.  In  the  case  a  sensor 
suite  exclusively  consists  of  identical  sensors,  one 
may  wonder  how  by  fusion  one  can  extract 
additional  qualitative  information  from  the  single 
sensors.  The  following  scenarios  can  be  thought  of 
to  illustrate  the  point: 

1.  Increase  in  spatial  resolution  via  synergy 
between  the  sensors:  By  suitable  sensor  setup 
and  associated  processing  a  synthetic  aperture 
can  be  formed  that  is  much  larger  than  the 
single-sensor  aperture. 

2.  Certain  NCTR  observations  (e.g.  jet  engine 
modulation  [REF])  are  confined  to  relatively 
narrow  solid  angles.  In  those  cases  it  can  be 
very  effective  to  have  multiple  sensors  located 
relatively  widely  apart  to  increase  the 
probability  of  detection. 

3.  During  tracking  multiple  same  sensors  may 
help  resolve  details  of  targets  flying  in 
formation,  or  identify  targets  that  vary  strongly 
in  aspect  dependent  on  their  orientation  (e.g.  a 
stealth  bomber  or  a  helicopter).  Multiple 
sensors  of  the  same  type  make  it  easier  to  do 
the  recognition  by  correlating  the  different 
aspects  of  the  target. 
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If  the  observables  of  the  fused  sensor  suite  are 
mutually  ‘orthogonal’,  complementary  fusion  will 
invariably  yield  more  information  than  each  of  the 
separate  sensors  can  provide.  It  is  therefore 
conceptually  the  simplest  way  to  demonstrate  the 
practical  benefits  of  fusion.  In  this  context  one 
could  make  an  analogy  between  a  single  sensor 
observation  of  the  real  world  as  a  (stochastic) 
projection  of  the  real  world  into  a  sensor 
observation  space.  In  this  analogy,  fusion  can  be 
seen  as  (partially)  reconstructing  the  real  world, 
representing  it  as  the  direct  product  space  of  all 
observation  spaces  of  sensors  that  participate  in 
the  fusion  suite.  Effectively  the  dimensionality  of 
the  observation  space  increases  by  adding  up  the 
dimensions  of  complementary  sensor  spaces. 
Adding  sensors  of  the  same  type  through  the 
Ergoden  hypothesis  basically  improves  the 
statistics  of  the  observation  in  the  particular  sensor 
space.  However  the  dimensionality  of  the  single 
sensor  space  does  not  increase  by  adding  more 
sensors  of  the  same  kind. 

In  the  case  that  the  observables  of  the  sensor  suite 
are  not  ‘orthogonal’,  the  fusion  process  can 
increase  the  speed  with  which  a  predefined 
accuracy  or  resolution  of  observation  is  achieved 
by  acting  as  a  smart  scheduler,  e.g.  via  cueing.  The 
shorter  response  time  is  realized  by  first 
determining  areas  of  interest  by  the  sensor  with  the 
lower  resolution  or  accuracy  and  the  largest  FOV, 
and  then  focusing  attention  on  these  areas  using 
the  high  accuracy  sensor,  instead  of  scanning  the 
entire  area  with  a  high  accuracy  sensor  with  a 
small  FOV.  In  addition  this  type  of  sensor  fusion 
increases  the  robustness:  if  the  cued  sensor  fails  or 
is  jammed,  the  other  one  can  take  over,  although 
with  lower  resolution  or  accuracy  (graceful 
degradation). 

An  example  of  multisensor  fusion  with  different 
sensors  is  the  combination  of  a  radar  measurement 
and  an  optical  image:  if  an  airplane  is  observed  by 
radar,  the  range  from  observer  to  the  airplane  is 
accurately  measured,  while  azimuth  and  elevation 
are  only  coarsely  determined.  In  contrast,  an 
optical  measurement  provides  azimuth  and 
elevation  with  relatively  high  accuracy,  whereas 
the  uncertainty  in  range  is  high.  Combination  of 
the  two  sensor  types  can  considerably  diminish  the 
absolute  uncertainty  in  the  position  of  the  airplane 
in  3D  space,  which  is  a  natural  consequence  from 


the  two  complementary  measurement  principles 
(see  Fig. 4). 

Finally  we  note  that  from  a  system  theoretical 
standpoint  the  observation  process  of  a  sensor  may 
be  viewed  as  a  projection  operator.  In  this  context 
sensor  fusion  holds  the  more  potential  the  earlier  it 
is  applied  (no  significant  reduction  of 
information).  Intuitively  we  can  express  the 
expected  effect  of  the  fusion  process  symbolically 
as: 

Si  ©  S2  >  Si  +  S2 

where  ©  represents  the  fusion  operator  and  S-,  is  a 
quality  measure  associated  with  sensor  i. 

3.5.3  Fusion  with  fuzzy  aggregation 
operators  as  a  way  to  reduce  complexity. 

In  our  study  we  will  concentrate  on  the  synergy  of 
sensors  at  the  signal  level  {“early  fusion”). 
Although  this  does  not  preclude  the  use  of  a  priori 
information,  or  taking  into  account  any  human 
generated  inputs  and  feedback,  we  focus 
specifically  on  the  sensor  area,  because  of  its 
fundamental  nature,  because  the  signals  are  not  yet 
distorted  or  corrupted  by  incoherent,  independent, 
or  ad  hoc  signal  analysis  operations,  and  because 
there  is  relatively  little  room  for  subjectivity.  The 
attractivity  of  this  approach  is  of  course  that  by 
operating  close  to  the  primary  sources  of 
information,  one  expects  to  be  able  to  significantly 
enhance  the  detection  and  recognition  processes  by 
applying  sensor  fusion. 

In  terms  of  the  functional  model  for  data  fusion  as 
proposed  by  the  Joint  Directors  of  Laboratories, 
we  will  be  exclusively  concerned  with  level  1 
processing.  It  should  however  be  pointed  out  that 
the  JDL  model  is  somewhat  artificial  and  in  our 
view  does  not  take  into  account  the  wealth  of 
possibilities  that  sensor  fusion  allows.  For  this 
reason  we  will  also  include  into  level  1  processing 
part  of  level  4,  dealing  with  real-time  sensor 
maintenance  or  sensor  scheduling.  It  is  absolutely 
necessary  to  include  real-time  feedback  aspects 
e.g.  the  on-line  calibration,  cueing,  and  sensor 
suite  optimization  in  case  of  damage. 

In  reviewing  data  fusion  literature  it  is  apparent 
that  the  vast  majority  of  the  research  is  concerned 
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with  level  2  and  level  3  processing,  dealing  with 
respectively  situation  and  threat  assessment.  This 
can  be  understood  in  the  sense  that  there  exists  a 
need  to  automate  the  decision  process  at  the  higher 
layers  and  that  it  is  necessary  to  frequently  interact 
with  humans  and  a  priori  knowledge.  However 
from  the  point  of  view  of  information  theory  it  is 
surprising,  because  it  is  obvious  that  the 
information  content  in  the  early  stages  of  data 
processing  is  higher  than  in  the  more  abstract, 
condensed  streams  of  inference  results  processed 
in  levels  2  and  3.  Even  in  level  1  processing  there 
are  more  papers  dealing  with  track  level  fusion 
compared  to  plot  level  fusion,  although  processing 
of  plots  would  be  preferred  being  earlier  than 
tracks.  Of  course  there  is  a  price  to  be  paid  for 
using  “raw”  sensor  data:  one  needs  considerable 
processing  power,  because  of  the  much  higher 
band  widths.  However  it  seems  that  this  is  not  the 
main  reason  for  the  interest  in  levels  2  and  3;  more 
researchers  seem  to  be  attracted  by  the  neatness  of 
well-structured  information-theoretical  data 
structures  instead  of  focusing  on  real-time 
processing  of  vast  amounts  of  data.  As  is  well- 
known  from  e.g.  high-energy  physics  experiments, 
such  situations  are  best  handled  by  constructing  a 
hierarchy  of  decision  levels  (‘trigger  hierarchy’), 
starting  from  relatively  crude  binary  decisions 
taken  in  very  short  time,  to  more  elaborate,  subtle 
inferences  on  the  resulting  data  stream. 

A  sensor  fusion  system  that  receives  raw  signal 
inputs  from  all  sensors,  retains  control  over  all 
primary  sensors  and  has,  at  least  theoretically,  a 
number  of  advantages  over  secondary  (level  2  and 
higher)  fusion.  Apart  from  the  larger  information 
content  of  raw  information,  it  should  however  be 
noted  that  each  fusion  step  requires  a  certain 
processing  time  and  that  in  early  fusion  it  is 
effectively  the  slowest  sensor  in  the  fusion  suite 
that  determines  this  latency,  even  if  we  neglect  the 
execution  time  for  the  fusion  process  itself.  In 
addition  the  latency  is  increased  because  fusing 
information  from  autonomous,  asynchronous,  and 
dissimilar  sensors  requires  synchronization.  A 
designer  of  a  sensor  fusion  testbed  should  be 
aware  of  this  and  take  precautions  to  ensure  that 
the  pileup  of  latencies  does  not  degrade  the  real¬ 
time  performance  of  the  fusion  system  as  a  whole, 
or  jeopardize  the  quality  of  the  fusion  process,  e.g. 
by  constructing  a  deficient  situation  awareness 
picture. 


It  is  relatively  straightforward  to  illustrate  the  idea 
of  sensor  fusion  by  the  improvement  of  operation 
of  a  target  tracker  during  the  observation  of  a 
maneuvering  target  in  cluttered  areas.  A  variety  of 
different  sensors  can  be  used  to  generate  plot 
reports  and  these  can  be  combined  on  the  basis  of 
a  simple  confidence  criterion,  in  turn  based  on  the 
presence  of  clutter  for  a  particular  sensor.  In  [21] 
this  has  been  illustrated.  However,  although  useful 
as  an  idea  this  example  basically  supports  the  idea 
of  increasing  robustness  by  increasing  the  number 
of  different  sensors.  Our  goal  is  more  ambitious: 
we  would  like  to  improve  the  quality  of  the  single 
sensor  conclusions  in  such  a  way  that 

P(S,vjS2)>P(Sx)+P{S2), 

or  if  this  condition  is  too  strong,  at  least 

P{SX  u52)  >  MAX(P(S,),P(S2)), 

where  P  (S,)  indicates  the  performance  or  ‘added 
value’  (the  effectively  information  accumulated 
over  time)  of  stream  Sj,  measured  by  sensor  i. 
From  this  formulation  it  is  clear  that  in  order  to 
model  sensor  fusion,  we  will  need  nonlinear 
operators. 

The  earliest  attempts  to  combine  measurements 
from  multiple  sources  is  by  Bayes  [22], 


There  are  a  number  of  difficulties  connected  with 
the  application  of  the  Bayesian  sensor  fusion 
formula: 

•  difficulty  in  assigning  a  priori  probabilities 

•  complexity  when  there  are  multiple  hypotheses 
and 1/  or  multiple  conditional  events 

•  requirement  that  hypotheses  have  to  be 
mutually  exclusive  and  exhaustive 

•  absence  of  uncertainty  modeling 

In  trying  to  find  an  appropriate  way  to  model 
fusion  in  the  presence  of  uncertainty  and  take 
advantage  of  the  nonlinearity  of  the  process, 
Dempster  created  a  generalization  of  Bayesian 
theory  that  allows  the  incorporation  of  uncertainty 
by  using  (overlapping)  probability  intervals  and 
uncertainty  modeling  to  determine  the  likelihood 
of  hypotheses  based  on  multiple  evidence.  The 
essential  generalization  of  Dempster-Shafer  (DS) 
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theory  is  that  not  all  hypotheses  need  to  be 
mutually  exclusive  as  in  the  Bayesian  theory.  In 
DS  fusion  an  evidence  interval  is  assigned  both  to 
single  and  to  more  general  propositions,  instead  of 
assigning  directly  a  probability  to  hypotheses  as  in 
Bayesian  theory. 

Evidence  in  DS  theory  is  modeled  based  on  the 
concept  of  probability  assignment  m(A),  a  function 
m:  p(X)  — >  [0,1],  where  £?(X)  is  the  power  set  of 
X,  m(0)=O  and 

Jm(A)  =  1 

Aep(X) 

For  each  set  A  e  p  ( X ),  m(A)  expresses  the 
proportion  to  which  all  available  and  relevant 
evidence  supports  that  a  particular  element  of  X 
belongs  to  the  set  A.  Although  this  normalization 
condition  resembles  a  similar  equation  for 
probability  density  functions,  there  is  a 
fundamental  difference:  probability  density 
functions  are  defined  on  X,  where  as  the  basic 
probability  function  is  defined  on  the  power  set 
p(X)  of  X. 

DS  evidence  theory  [23]  has  a  clear  advantage 
over  the  Bayesian  formulation  that  it  allows  for 
uncertainty  in  the  form  of  conflicting  evidence  by 
adopting  belief  and  plausibility  measures  that 
unlike  probability  measures  are  no  longer  additive. 
Instead,  weaker  conditions  hold  for  these  measures 
as  expressed  by: 

Bel(A]  u  A2u . uA„)> 

JjBel(Aj)-^Bel(AjnAk)  + 

J  j<k 

+ . (-l)"+l  Bel(A:  n  A2n . nAJ 

and 

//(A,  n  A2n . nAn)< 

JjPl(AJ)-JjPl(AjuAk)  + 

j  j<k 

+ . (-1  )"+,P/(A1  uA2u . uAJ 

where  the  belief  and  plausibility  measures,  Bel(A) 
and  Pl(A),  respectively,  are  defined  by  the  basic 
probability  assignment  function  m  by: 

Bel(A)  =  ^m(B) 

BqA 


Pl(A)  - 

Ar\B*0 

From  the  definition  it  is  obvious  that  /7(A)  > 
Bel(A).  the  duality  of  the  two  measures  is  given 
by: 

/7(A)  =  1  -  Bel(A) 

Although  the  probability  measure  Prob(A)  is  not  a 
fuzzy  measure,  it  may  be  seen  as  a  limiting  case  of 
belief  and  probability  measures: 

Bel(A )  <  Prob(A)  <  //(A) 

In  the  DS  approach  Bel  (sometimes  called 
‘support’)  and  PI  are  interpreted  as  lower  and 
upper  probabilities,  respectively,  or  alternatively 
as  the  boundaries  of  the  evidencial  interval.  From 
the  definition  of  the  belief  and  probability 
measures  the  following  identities  hold: 

Bel(X)  =  1  and  /7(0)  =  0 

These  identities  reflect  that  total  ignorance  may  be 
expressed  by  Bel(X)  =  1  and  Bel(A?X)  -0  for  all 
Ae  p(X),  or,  alternatively,  by  Pl(0)= 0  and 
Pl(A^0)  =1  for  all  Ae  In  DS  theory  the 

combined  evidence  m\i2  from  two  sensors  (1  and  2) 
is  expressed  as: 

JmI(5)m2(C) 

(A)  =  anc^ -  for  A  ^  0  and 

1  -K 

mi2(0)  =  0 

where  K  =  ^m,(/)m2(C). 

5nC=0 

The  renormalization  term  (1-K)  follows  from  the 
condition  that  m]t2  must  again  be  a  basic 
probability  assignment,  i.e.  ^ m,  2  (A)  =  1 ,  and 

Acp(X) 

reflects  the  handling  of  conflicting  evidence.  K 
may  be  interpreted  as  a  measure  of  the  resulting 
uncertainty  after  fusion. 

Noting  that  belief  and  plausibility  measures  are 
both  examples  of  Sugeno’s  /.-fuzzy  measure  g*, 
[24],  the  question  arises  whether  it  is  possible  to 
combine  the  intuitive  ideas  on  sensor  fusion  and 
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the  properties  of  gx ,  we  willl  argue  that  in  contrast 
the  basic  probability  assignment  in  DS  theory, 
fuzzy  gx  measures  can  be  tuned  to  the  problem 
consideration. 

We  will  take  a  closer  look  at  this  in  the  following 
and  propose  to  view  the  multisensor  fusion  process 
in  terms  of  a  synergy  between  (sets  of)  sensors  that 
are  grouped  in  such  a  way  as  to  support  a  certain 
decision  or  hypothesis.  Instead  of  attempting  to 
make  a  decision  (detection  or  classification)  in  one 
step,  either  by  a  single  sensor,  or  by  a  linear 
combination  of  a  group  of  sensors,  it  is  proposed 
to  combine  supporting  evidence  for  a  hypothesis  in 
a  hierarchical  way  by  building  a  tree  structure  that 
combines  at  the  lowest  level  clusters  and  in  the 
next  levels  combines  the  outputs  of  several  initial 
cluster  in  superclusters  and  so  on. 


importance  g  towards  the  final  decision  become 
additive  and  coincides  with  the  definition  of  a 
probability  measure. 

Following  ideas  put  forward  in  [25],  sensor  fusion 
may  be  modelled  using  the  concept  of  fuzzy 
integration.  For  a  review  on  the  role  fuzzy 
integrals  in  the  framework  of  multiple  criteria 
decision  making  see  [26].  A  fuzzy  integral  may  be 
interpreted  as  an  aggregation  functional  of 
subjective  evidence,  where  the  subjectivity  is 
expressed  in  the  fuzzy  measure,  and  integation  is 
defined  over  measurable  sets  [27],  In  contrast  to 
normal  (Lebesque)  integrals,  fuzzy  integrals  are 
non-linear  functionals.  It  is  exactly  this 
nonlinearity  and  the  possibility  to  include  a  fuzzy 
measure  g\  that  is  attractive  in  the  context  of 
fusion. 


At  each  level  in  the  tree  decisions  need  to  be  made 
from  different  sources  with  different  importance. 
This  is  conveniently  modeled  by  the  Fuzzy  X- 
measure  g\  (0<  g>_  <1).  In  particular  we  have  in  the 
absence  of  relevant  information  towards  the 
classification/detection  goal:  g(0)  =  0  and 
g{A)  <  g(B )  if  Acz  B  .  This  coincides  with 
the  intuitive  feeling  that  if  the  evidence  support  is 
larger  ( i.e .  if  we  observe  the  same  scene  with  more 
sensors.),  that  then  the  information  content  should 
also  increase.  In  addition  the  following  property 
holds  for  all  A,B  c  X  with  A  n  B  =  0  : 


Formally  Sugeno’s  fuzzy  integral  is  defined  as 
[24]: 


fuzzy, A 


h(x)°g(*)  =  sup 


min 


min(/r(x),g(A  n  E)) 

V  j 


Hence  the  evaluation  of  the  fuzzy  integral  may  be 
interpreted  as  evaluating  the  degree  of  agreement 
between  objective  evidence  h(x)  and  the 
expectation. 

We  will  not  discuss  the  properties  of  this  fuzzy 
fusion  operator  here,  but  note  that  it  is  ideally 
suited  to  combine  information  of  different  sources 
without  the  risk  of  the  combinatorial  explosion. 


3A>-1  g(AyjB)  =  g(A)  +  g(B)  +  Xg(A)g(B) 

This  again  supports  the  intuition  that  adding  more 
independent  data  ( A  n  B  =  0 )  cooperates 
towards  an  increase  in  confidence  about  the  final 
decision.  In  addition  both  intuitive  features  about 
the  fusion  of  two  independent  sensors  are 
reproduced,  viz. 

A>0  g(A u B)  >  g(A)  +  g(B)  ,  i.e.  fusion 
is  more  than  superposition  and 

-1  <  A  <  0  g(AuB)>MAX(g(A),g(B)), 
implying  that  even  when  the  level  of  confidence  is 
larger  than  1  (as  reflected  by  the  negative  X)  then 
still  it  may  be  fruituitous  to  apply  sensor  fusion.  In 
the  event  that  1=0,  i.e.  the  case  where  all  sensors 
have  the  same  importance  and  complexity  covers 
the  whole  universe  of  discourse,  the  degree  of 


A  similarity  between  fuzzy  fusion  (FF) 
aggregation  and  the  way  DS  theory  fuses  data 
from  different  sources  is  that  both  make  use  of 
fuzzy  measures:  DS  uses  the  belief  measure 
exclusively,  whereas  the  FF  operator  uses  the  gx 
measure.  For  X  >  0  this  measure  is  equivalent  with 
the  DS  belief  measure).  The  conceptual  difference 
between  both  methods  is  twofold: 

1.  the  frame  of  discernment  (or  universe  of 
discourse)  is  different 

2.  clear  separation  of  objective  and  subjective 
uncertainty  in  the  case  of  FF 

We  will  illustrate  these  points  in  the  following: 
For  the  FF  scheme  the  frame  of  discernment 
contains  the  information  sources  (the  sensors) 
related  to  the  hypothesis  under  consideration, 
whereas  in  DS  theory  the  universe  of  discourse 
contains  all  possible  hypotheses.  In  combining  the 
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different  information  streams,  the  fuzzy  fusion 
aggregator  fuses  all  sources  according  to  their 
relative  a  priori  importance  as  well  as  to  the  degree 
to  which  each  sensor  supports  the  hypothesis 
under  consideration.  In  contrast,  the  fusion  process 
in  DS  theory  associates  with  each  knowledge 
source  a  belief  function  that  is  defined  over  the 
power  set  of  the  set  of  hypotheses  and  combines 
these  in  the  fusion  process.  The  evaluation  of 
Dempster's  rule  of  combination  therefore  has 
exponential  complexity  0(2N),  where  N  equals  the 
number  of  hypotheses  under  consideration.  In 
contrast,  in  FF  one  fuzzy  integration  has  to  be 
calculated,  which  implies  that  g\  has  to  be 
calculated  nN  times,  where  n  is  the  number  of 
sensors.  The  evaluation  of  the  fuzzy  integral  can 
then  be  carried  out  in  O(n)  steps.  The  second 
advantage  of  fuzzy  aggregation  is  that  both  the 
weighting  with  the  degree  of  support  by  which  a 
sensor  supports  a  certain  hypothesis,  but  also  the 
weight  of  importance  of  a  certain  sensor, 
reflecting  a  subjectivity  or  a  a  priori  confidence  in 
the  particular  sensor. 

We  therefore  conclude  that  the  formalism  of  fuzzy 
measure  theory  may  offer  an  opportunity  to  model 
the  process  of  sensor  fusion  in  a  natural,  intuitive 
and  adequate  way,  allowing  arbitrary  sensors  to  be 
fused  and  allowing  different  ways  to  weigh 
different  combinations  of  observations  (see  Table 


2).  The  example  reflects  the  property  that  if  sensor 
2  ("the  most  decisive  sensor  in  support  of  the 
hypothesis  under  consideration  of  the  three 
sensors")  is  combined  with  one  of  the  other  two 
sensors  (1  or  3),  the  combined  evidence  as 
reflected  by  g\  must  be  larger  than  the  sum  of 
g2+g3.  This  is  indeed  the  case,  as  we  see  from  the 
example:  g2  ©  g3  =  g23  =  0.687  >  0.5  =  g2  +  g3  and 
also  g23  >  g2i  >  gi3,  which  we  would  also  expect 
intuitively  on  the  basis  of  fusion  as  a  nonlinear 
operation  with  positive  synergy. 

Table  2.  Example  of  a  fuzzy  gx  measure  with 
X  >  0  (belief  measure). 


Member  of 
Powerset  p(A) 

Belief  measure 
g?.(A)  %  =  3.109 

0 

0 

{Xl} 

0.1 

{x2} 

0.3 

{x3} 

0.2 

{Xl  ,  x2} 

0.493 

{Xl  ,  x3} 

0.362 

{x2,x3} 

0.687 

{Xl  ,  x2  ,  x3}=X 

1.0 

Figure  7  The  architecture  of  the  SFD. 
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4  IMPLEMENTATION  OF  THE  SFD 

4.1  System  architecture 

Multisensor  fusion  can  be  defined  as  the  study  of 
concepts  and  techniques  developed  for  optimal 
information  processing  in  a  generally  distributed 
multisensor  environment  through  a  clever 
integration  of  sensor  data  streams. 

We  will  limit  us  here  and  consider  only  fusion  at 
the  lowest  level,  i.e.  fusion  of  raw  sensor  data, 
possibly  after  some  preprocessing,  but  excluding 
any  combination  with  other  sensor  streams  or  large 
database  management  systems  and  other  abstract 
(non-physical)  information  streams.  Although  the 
final  goal  is  to  do  sensor  fusion  in  real  time,  the 
initial  goal  is  to  select  and  develop,  test  and 
evaluate  suitable  sensor  fusion  algorithms.  The 
minimum  real-time  demand  on  the  sensor  fusion 
testbench  therefore  is  that  as  much  as  possible  raw 
data  can  be  stored  on  disk  in  real  time.  Even  this 
rather  obvious  demand  is  in  practice  not  so  trivial. 

The  option  of  data  reduction  via  filtering,  e.g  by 
MPEG  or  JPEG  encoding  schemes  can  be  ruled 
out.  The  fact  that  video  signals  generated  by 
commercial  CCD  cameras  are  interleaved  for  the 
comfortable  perception  of  the  observer,  is  in  terms 
of  data  acquisition  an  undesirable  feature  that 
either  halves  the  vertical  image  resolution  or 
displays  artifacts,  corresponding  to  the 
superposition  of  two  half-resolution  frames 
captured  at  different  times.  This  becomes  e.g. 
visible  in  the  case  of  fast  moving  targets  close  to 
the  camera.  Compression  techniques  are  only 
allowed  if  they  can  be  applied  in  real  time  and  if 
they  can  guarantee  to  be  invertable  (lossless 
compression). 

Because  raw  sensor  data  still  have  maximum 
information  content,  they  are  best  suited  for 
studying  data  fusion.  At  this  level  one  also  has  the 
maximum  data  rates.  During  the  conceptual  design 
phase  it  was  thought  useful  to  design  a  data- 
acquisition  system  that  should  be  capable  of 
digitizing  the  inputs  from  four  optical  cameras, 
and  a  radar  system.  The  radar  signals  should  be 
stored  with  an  effective  bandwidth  of  20  MHz, 
corresponding  to  an  instantaneous  data  rate  of  40 
Msamples/s  @  12  bit  and  a  sustained  rate  of  20 
MB/s.  A  short  survey  of  available  commercial-of- 


the-shelf  hardware  indicated  that  AD  conversion 
for  radar  can  be  done  and  that  also  frame  grabbers 
for  the  CCD  cameras  are  readily  available.  It 
seemed  initially  also  possible  to  meet  the  required 
sustained  data  rate  of  approximately  48  MB/s  for 
the  datastream  of  all  cameras  and  a  similar  data 
rate  for  the  radarvideo  signals.  Apart  from  coping 
with  these  massive  data  streams,  the  signals  must 
also  be  labeled  in  real  time:  the  frames  from  the 
different  cameras  and  each  radar  echo  need  to  be 
timestamped  before  writing  the  signal  to  disk.  A 
typical  scene  lasts,  say,  5  minutes,  so  that  if  we 
want  to  record  3  sequences  we  would  need  in  total 
about  50  GByte  storage,  requiring  a  redundant 
array  of  “inexpensive”  disks  (RAID).  In  the  next 
step  the  measurements  are  stored  on  magnetic 
tape. 

Both  for  practical  as  well  as  for  storage  reasons  the 
data-acquisition  system  will  be  built  around  two 
powerful  PCI-bus  systems,  each  equipped  with  its 
own  RAID  storage  system.  The  PCI-bus  is 
specified  up  to  a  sustained  transfer  rate  of  70 
MB/s,  so,  apart  from  some  operating-system 
overhead,  no  throughput  difficulties  are  to  be 
expected.  For  reasons  of  stability  the  Windows  NT 
vs.  4.0  operating  system  was  selected,  which 
allows  interference  of  the  user  in  defining  task 
priorities  so  that  the  full  PCI-bus  capacity  can  be 
dedicated  to  the  data-acquisition  process. 

The  architecture  of  the  SFD  is  shown  in  Fig.  7  for 
the  case  of  4  cameras  (2  CCD’s,  2  IR  FPA’s)  and 
a  pulsed  radar.  Camera  outputs  may  be  either 
analog  or  digital,  may  be  free-running  or  locked  to 
each  other  and  the  radar  sensor  may  be  either 
staring  or  rotating.  After  AD  conversion  signals 
are  time-stamped  and  stored  on  two  RAID  mass 
storage  devices,  together  with  the  azimuth  angle  of 
the  radar  antenna.  Care  is  taken  to  maintain  the 
full  dynamic  range  of  the  signals  as  much  as 
possible,  without  reducing  the  original  bandwidth. 
Absolute  time  is  stored  with  each  sample,  i.e.  for 
the  cameras  with  each  frame  and  for  the  radar  per 
pulse  return  (time  histogram).  In  this  way  the 
relative  order  of  the  various  streams  is  ensured. 
The  time  stamp  is  derived  from  an  IRIG-B  coded 
timing  signal  produced  by  a  GPS  receiver  and  has 
a  resolution  of  1  ms.  In  order  to  guarantee  also  a 
unique  ordering  of  radar  histograms  at  the 
maximum  PRF  of  4  kHz,  we  would  need  a 
resolution  of  250  ps.  Special  provisions  have  been 
devised  to  ensure  that  even  at  the  highest  PRF 
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unique  ordering  is  achieved  despite  the  lower  time 
resolution  of  IRIG-B.  Although  IRIG-B 
synchronizes  with  absolute  time  (UTC),  only 
relative  timing  is  important,  because  the  SFD  is 
intended  for  use  on  the  same  platform,  with  the 
sensors  positioned  close  together. 

In  case  of  a  scanning  radar  antenna  we  store  in 
addition  to  the  proper  sensor  signals  also  the 
azimuth  and  elevation  angles  with  their  own  time- 
stamp.  Due  to  external  influences,  the  scanspeed 
will  in  general  not  be  constant,  so  that  the 
sampling  of  the  azimuth  angle  is  not  equidistant  in 
time. 

The  data-acquisition  system  is  currently  under 
development  in  close  cooperation  with  a  system 
integrator.  It  has  been  shown  that  despite  the 
advertised  specifications  given  in  the  data  sheets 
for  e.g.  commercial  RAID  systems,  the  actual 
system  performance  in  terms  of  sustained  data 
rates  as  guaranteed  by  harddisk  manufacturers  is 
down  by  at  least  a  factor  of  3,  transferring  at 
maximum  at  a  data  rate  of  about  12  MB/s.  Further 
investigation  has  shown  that  the  solution  of  the 
storage  problem  needs  the  investment  of  a 
considerable  amount  of  time  into  this  problem  in 
absence  of  expert  support  by  the  equipment 
manufacturers.  Real-time  standards  are  absent, 
data  sheets  vague  or  simply  wrong  and  in  one  case 
it  was  actually  found  that  the  manufacturer 
published  transfer  rate  performance  data  measured 
with  a  tool  of  unknown  origin.  We  are  by  now 
convinced  that  the  required  data  transfer  rates  can 
indeed  be  realized.  Attaining  the  necessary 
performance  however,  may  in  the  end  come  down 
to  developing  a  new  hardware  interface  board, 
including  the  development  of  a  new  PCI  chipset  to 
obtain  the  maximum  PCI-bus  performance,  as  well 
as  the  development  of  new  drivers  for  industrial 
standards  under  Windows  NT.  This  experience 
illustrates  that  there  is  still  a  long  way  to  go  in 
developing  real-time  data-acquisition  systems  with 
commercial-of-the-shelf  products. 

In  conclusion  it  is  clear  that  although  ideal 
standards  for  data  acquisition  are  not  yet 
commercially  available  and  universal  libraries  for 
hardware  and  software  for  data  acquisition  do  not 
yet  exist,  it  is  still  possible  to  build  a  cost-effective 
system  out  of  commercial  available  components 
that  does  the  job,  provided  that  one  keeps  in  mind 
what  the  consequences  of  this  suboptimality  are 


for  the  final  system  performance.  Some  of  the 
functional  requirements  that  have  been  outlined 
above  may  be  realized  with  VMEbus  systems,  but 
at  a  much  higher  price.  At  the  system  software 
level  the  situation  is  more  alarming:  effective  real¬ 
time  operating  systems,  with  a  good  GUI,  are  hard 
to  find  and  real-time  drivers  for  the  boards  that  one 
wishes  to  combine  are  seldom  available. 
Alternatively,  the  driver  that  one  buys  with  a  board 
appears  to  be  8-bit,  whereas  the  ADC  on  the  board 
delivers  12  bits.  If  maximum  performance  is 
required  one  should  be  prepared  to  invest  time  in 
the  problem  oneself  or  work  together  with  an 
expert  system  integrator.  At  the  moment  it  is  the 
only  way  that  will  solve  hard  real-time  data- 
acquisition  problems. 

In  Fig.  7  an  optional  preprocessing  process  is 
shown.  Although  initially  data  will  be  analyzed 
and  algorithms  developed  off-line,  on  a  different 
computer  platform,  this  option  makes  it  possible  to 
test  fusion  algorithms  in  real  time  by  inserting 
extra  processing  capability.  To  this  end  SHARC 
DSP  boards  can  be  installed  in  the  SFD  that  add 
scaleable  processing  power  to  the  SFD. 

Via  the  system  software  it  is  possible  to  get  a  first 
impression  of  the  data  during  storage  by  selecting 
regions  of  interest  in  the  radar  stream  by  shifting  a 
delayed  time  window  relative  to  the  transmitter 
pulse. 

A  standard  magtape  drive  is  used  for  backup  of 
data  from  the  RAID  sytem.  After  completing  the 
measurement  the  user  writes  selections  of  data  to 
tape  as  metafiles,  i.e.  in  an  intermediate 
representation,  which  enables  him  to  store  these 
metafiles  from  the  RAID  system  to  tape  using  the 
Windows  NT  operating  system  and  to  read  these 
metafiles  back  to  a  SUN  workstation  under  Solaris 
for  analysis  and  vice  versa. 


4.2  Model  experiment 

In  defining  a  model  experiment  to  test  the 
functionality  of  fusion  algorithms  a  number  of 
possible  relevant  model  experiments  have  been 
considered.  Suitable  experiments  are  such 
experiments  that  try  to  observe  or  detect  targets 
that  are  hard  to  detect  by  individual  single  sensors 
and  that  one  can  hope  for  that  sensor  fusion  could 
offer  a  solution.  It  should  be  noted  that  there 
should,  at  least  theoretically,  be  a  reasonable 
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probability  of  detection  for  the  sensors  that 
participate  in  the  fusion  process,  because 
otherwise  there  is  not  much  point  in  attempting  the 
fusion,  as  we  have  seen  in  Sec.  3.2.  The 
dissatisfactory  performance  of  single  sensors  may 
be  due  to  our  inability  to  attach  a  (preferably 
scene-based)  correct  confidence  level  to  the 
observation  under  a  vast  range  of  changing 
environmental  conditions,  or,  alternatively,  it  may 
be  due  to  resolution  and  visiting  rate  of  a  sensor  in 
combination  with  an  insufficient  modeling  of 
clutter,  even  under  well-defined  circumstances.  In 
the  first  case  we  have  essentially  a  scheduling 
problem:  Which  sensor  in  the  suite  is  optimal 
given  the  circumstances?  or  Which  sensor  can  cue 
the  other  sensor(s)  given  a  particular  observation 
task?  In  the  second  example  effort  must  be 
invested  in  analyzing  the  detection  steps  and  the 
validity  of  the  clutter  models  used  therein.  Two 
experiments  have  been  considered  in  some  detail: 
Detection  of  buried  anti-personnel  landmines 
(APM)  and  the  detection  and  identification  of 
small  sea  targets  in  a  wide  variety  of 
circumstances.  Small  targets  means  small  patrol 
boats,  dinghies,  sea  mines,  the  head  of  a  frogman, 
buoys,  debris,  etc.  Because  of  the  still  ongoing 
developments  of  constructing  suitable  sensors  for 
mine  detection,  it  has  been  decided  to  attack  first 
the  problem  of  detecting  small  targets  at  sea  with 
the  SFD. 

It  is  expected  that  a  combination  of  visible  light 
and  IR  sensors  together  with  radar  must  be  capable 
of  detecting  small,  floating  objects.  Sensor  fusion 
is  expected  to  play  an  important  part  in  the 
detection  and  recognition  of  small  sea  targets 
because  small  sea  objects 

•  have  both  a  small  optical  cross  section  and  a 
small  RCS  (radar  cross  section) 

•  are  not  permanently  observable  (this  is 
especially  relevant  for  scanning  sensors), 
depending  on  their  height  and  the  sea  state 

•  have  almost  the  same  temperature  as  the 
surrounding  sea  water 

•  have  a  low  velocity  relative  to  the  water,  so  that 
effective  clutter  suppression  becomes  difficult 

•  must  be  detected  in  a  sea  clutter  spectrum  for 
radar  and  electro  optics  that  is  complicated 
because  of  its  time  variance  and  its  spatial  and 
temporal  coherence. 


Parallel  to  the  construction  of  the  SFD  a  research 
effort  has  been  started  to  make  an  inventory  on 
radar  sea  clutter  models  and  their  validity. 


5  CONCLUSIONS 

In  this  article  we  presented  early  sensor  fusion  as  a 
possible  way  to  demonstrate  the  advantages  of 
sensor  synergy.  Early  sensor  fusion  enables  us  to 
maximally  benefit  from  sensor  observations,  but 
requires  at  the  same  time  extensive  data 
acquisition  efforts.  Sensor  fusion  is  motivated  by 
the  expected  qualitative  and  quantitative 
improvement  of  observations  and  thus  of  situation 
awareness.  In  order  to  test  and  verify  these 
benefits  experimentally  in  a  quantitative  way,  a 
sensor  fusion  demonstrator  testbed  (SFD)  has  been 
designed  and  is  presently  under  construction  at 
TNO-FEL.  The  system  architecture  and  its  design 
philosophy  have  been  outlined.  The  real-time 
constraints  and  the  need  to  (re)use  intermediate 
fusion  results  in  different  decision  processes 
suggest  that  in  early  fusion  soft  decisions  are  more 
effective  than  hard  decisions.  Finally  it  can  be 
concluded  that  from  the  theoretical  point  of  view 
key  concepts  of  fuzzy  logic,  such  as  fuzzy 
measures  and  Sugeno’s  fuzzy  integral,  provide  us 
with  suitable  tools  to  combinevery  different  sensor 
streams  with  soft  decisions  that  show  the  type  of 
synergy  behavior  expected  of  a  fusion  process. 
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PAPER  No:  31 


DISCUSSOR’S  NAME:  E.  Bosse 


COMMENT/QUESTION: 

-  First  observation:  By  trying  to  fuse  data  at  the  lowest  possible  level  you  are  imposing  a 
constraint  of  similarity  among  the  sensors. 

-  Question:  In  the  voting  algorithm  used  for  the  fusion  of  n  identical  sensors,  do  you  consider  any 
weighting  function  to  reject  bad  quality  sensor  information?  If  not,  it  may  be  dangerous  to 
conclude  that  sensor  fusion  can  deteriorate  the  end  product.  In  any  sensor  fusion 
implementation,  sensor  information  should  be  weighed  according  to  its  quality. 


AUTHOR/PRESENTER’S  REPLY: 

-  Yes,  this  is  absolutely  correct:  we  intend  to  demonstrate  that  sensor  fusion  is  not  always 
advantageous.  For  this  reason,  a  simplified  example  of  N  (N  odd)  same  copies  of  sensors  is 
taken  to  make  sure  that  the  results  are  only  dependent  on  the  single-sensor  possibility  of 
detection  (and  of  course  on  the  decision  method,  in  this  case  majority  voting. 

-  Answer  to  the  Question 

No,  we  did  not  apply  any  weighing  in  the  process,  for  the  same  sensors  as  formulated  above.  For 
the  sake  of  simplicity  we  assume  that  the  sensors  are  perfectly  the  same  and  no  sensor  is 
malfunctioning.  Of  course,  it  is  always  possible  to  take  this  type  of  behaviour  into  account,  but  it 
only  complicates  the  argument.  However,  it  can  easily  be  seen  that  including  mulfunctioning 
sensors  will  at  best  exclude  these  sensors  from  the  fusion  process  and  therefore  the  fusion 
process  reduces  again  to  an  effectively  smaller  N  -  fusion,  i.e.  no  improvement  by  adding  more 
sensors.  In  case  they  are  not  rejected,  the  fusion  process  will  only  deteriorate  even  further.  All 
the  above  conclusions  are  valid  under  the  assumption  of  using  positive  weights  only. 
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PAPER  No:  31 

DISCUSSOR’S  NAME:  P.M.  Zanker 

COMMENT/QUESTION: 

What  plans  do  you  have  to  construct  a  test  bed,  and  what  form  will  this  take? 


AUTHOR/PRESENTER’S  REPLY: 

The  testbed  will  primarily  be  concerned  with  designing  the  hardware  on  the  level  of  sensor  fusion. 
The  initial  stage  will  be  concerned  with  storage  of  new  data  to  facilitate  design  and  testing  of 
fusion  algorithms  (on  different  platforms  ....  SUN,  PC  etc.).  Finally,  the  most  suitable  algorithms 
will  be  implemented  in  hardware  preprocessing  units  that  add  scaleable  (DSP)  processing  power 
to  the  testbed. 
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1.  SUMMARY 

The  NEMESIS  programme  at  DERA  Malvern  was 
instituted  to  investigate  aspects  of  multi-sensor,  single 
node  data  fusion  applied  to  the  task  of  air  target 
identification  (ID)  in  the  military  environment.  In 
recent  years,  a  reduction  of  force  numbers  combined 
with  the  introduction  of  more  capable  weapon  systems, 
such  as  precision  guided  weapons  and  beyond  visual 
range  missiles  has  made  the  requirement  for  accurate, 
timely,  reliable  and  robust  target  identification  a  high 
priority  for  air  defence  forces.  After  much  research 
during  the  last  25  years  into  ID  technologies  and 
techniques,  it  is  generally  conceded  that  no  single 
identification  technology  or  system  can  now  nor  will,  in 
the  foreseeable  future,  satisfy  all  the  operational 
capabilities  and  performance  requirements  needed  from 
air  target  identification  systems.  The  enforced  use  of 
several  identification  sensors  and  sources  implies  the 
need  for  the  fusion  of  the  ID  data  derived  from  these 
different  sensors  and  sources.  Studies  have  indicated 
that  automated  ID  fusion  is  likely  to  provide  accuracy, 
consistency,  timeliness  and  manpower  benefits.  The 
NEMESIS  programme,  sponsored  by  the  UK  MoD,  has 
investigated  the  application  of  a  dedicated  NATO  air 
target  ID  data  fusion  algorithm,  NATO  STANAG  4162 
(see  reference  [1]),  to  a  multi-sensor,  single  node 
SHOrt  Range  Air  Defence  (SHORAD)  demonstrator 
system.  This  paper  describes  the  demonstrator  package, 
the  sensors,  the  fusion  process  and  the  results. 

2.  INTRODUCTION 

Although,  by  1993,  much  theoretical  development  and 
simulation  of  STANAG  4162  effectiveness  had 
occurred,  the  UK  MoD  required  a  greater  insight  into 
the  practical  difficulties  of  applying  the  Identification 
Data  Combining  Process  (IDCP),  as  described  in 
STANAG  4162,  to  a  real  air  target  tracking  and  ID 
system.  The  solution  chosen  used  several  sensor  and 
source  packages  which  reflected  current  practices  in 
regard  to  aircraft  ID,  which  were  suitable  (and 
available)  for  aggregation  in  a  practical  demonstrator, 
and  which  encompassed  the  wide  range  of  different 
sensors  and  sources  detailed  in  STANAG  4162  Annex 
D.  The  demonstrator  brought  together  many  techniques 


ranging  from  track  correlation,  ID  association,  fusion, 
Human  Computer  Interface  displays  (HCIs),  to  the 
handling  of  large  quantities  of  air  target  library  data  and 
innovative  visual  identification  processes.  NEMESIS, 
once  developed,  was  used  during  a  later  phase  to  gather 
target  data  for  the  extraction  of  performance  metrics  to 
gain  insight  to  the  perceived  benefits  from  this  form  of 
ID  fusion. 

3.  STANAG  4162 

STANAG  4162  describes  in  detail  the  IDCP  and  the 
recommended  manner  in  which  it  should  be 
implemented.  It  has  been  in  development  for  some 
years  and  the  concept  is  now  accepted  by  the  major 
NATO  nations  who  continue  to  support  its  further 
development.  This  specific  method  of  Bayesian  fusion 
is  currently  being  employed  for  the  ID  function  in  at 
least  one  major  NATO  programme  and  one  UK 
significant  procurement  programme.  Other  NATO 
nations  are  also  known  to  be  carrying  out 
demonstrations  with  the  concept.  It  is  important  that 
nations  continue  to  employ  the  STANAG  due  to  the 
obvious  inter-operability  benefits  that  will  accrue  from 
ID  data  exchange  in  the  very  difficult  ID  task. 

4.  DESCRIPTION  OF  THE  NEMESIS 
DEMONSTRATOR 

NEMESIS  consists  of  a  radar  system  supported  by  a 
range  of  sensors  designed  to  use  specific  aircraft 
characteristics  such  as  shape,  emissions,  position  etc  to 
identify  air  targets.  The  track  forming  systems  for 
NEMESIS  are  an  I  band  primary  radar  and  a  secondary 
surveillance  radar  deriving  target  tracks  from  aircraft 
transponders,  where  they  are  fitted.  Fundamental  to 
STANAG  4162  fusion  principles  is  the  requirement  for 
a  target  track  to  exist  prior  to  the  attachment  of  any  ID 
data.  The  ID  sensor  package  was  ambitious  (in  order  to 
test  as  many  as  possible  of  the  STANAG  Annex  D 
options)  and  incorporates  a  wide  range  of  disparate 
sensors  ranging  from  acoustic  and  IFF  to  thermal  and 
visual  identification.  The  conversion  of  source  and 
sensor  declarations  to  a  common  format  suitable  for 
algorithmic  fusion  was  an  involved  and  complex 
process.  The  radar,  sensors  and  sources  are  distributed 
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between  three  cabins  and  two  vehicles  all  linked  by 
fibre  optic  data  and  communications’  networks.  The 
central  data  fusion  cabin  also  provides  the  control 
functions  for  directed  sensors  and  management  of  all 
library  and  analysis  processes.  The  demonstrator 
handles  a  maximum  of  50  air  tracks  and  can  provide 
real-time  ID  recommendations  of  target  allegiance 
within  a  few  seconds.  To  permit  additional  study  to  be 
performed  in  the  absence  of  a  live  environment,  a 


simulation  capability  is  built  into  the  demonstrator  to 
generate  the  conditions  that  do  not  occur  naturally  in 
peace  time.  The  HCI  consists  of  two  Sun  workstations, 
configured  for  system  management,  display  of  the 
recognised  air  picture,  sensor  control  and  analysis. 
Access  to  all  stages  of  the  data  fusion  process  are 
available  through  various  displays.  The  NEMESIS 
system  layout  is  shown  in  Figure  1. 
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FIGURE  1:  THE  NEMESIS  SYSTEM  LAYOUT 


5.  THE  SENSOR  PACKAGE 

The  sensor  and  source  package  used  for  deriving 
aircraft  allegiance  consists  of  the  following  blend  of 
co-operative  and  non  co-operative  identification 
systems: 

•  Identification  Friend  or  Foe  -  this  military 
standard  system  comprising  Mark  10a  and  Mark 
12  IFF  equipments  derives  identification  by  a 
question  and  answer  routine  with  co-operating 
aircraft,  some  aspects  of  which  are  secure. 

•  Electronic  Support  Measures  (ESM)  -  this  non  co¬ 
operative  identifier  detects  RF  transmissions  over 
a  broad  spectrum  and  classifies  the  target 
emitter(s)  by  reference  to  a  look  up  table  of  library 
data.  In  some  cases  emitter  ID  can  be  mapped  (see 
section  7)  to  platform  type  and  allegiance. 


•  Visual  identification  -  this  auto-tracking  dual 
(wide  and  narrow)  field  of  view  camera  system, 
uses  a  man-in-the-loop  to  make  judgements  of 
target  type  based  upon  his  training  and  experience 
of  platform  images.  The  allegiance  of  the  target  is 
then  linked  via  library  data  to  the  platform  type, 
and  allowance  is  made  in  the  mapping  process  for 
occasions  where  both  own  forces  and  enemy 
forces  operate  the  same  platform  type. 

•  Thermal  identification  -  This  is  a  similar  process 
to  the  visual  function  but  operating  in  the  8-12 
micron  infra-red  band  with  a  Thermal  Imager 
Common  Module  (TICM)  system.  This  imager 
extends  the  platform  identification  function 
through  the  hours  of  darkness. 

•  Acoustic  detection  -  This  sensor,  comprising  a 
linear  array  of  microphones,  is  dedicated  to  the 
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identification  of  helicopter  platforms  and  provides 
platform  type,  range  and  bearing  by  comparison 
of  a  noise  signature  with  a  signature  library. 

•  Airspace  Control  Procedures  (ACPs)  -  With  prior 
knowledge  of  flight  corridors,  safe  lanes  and 
mission  plans,  the  tracks  of  aircraft  crossing  the 
area  of  interest  can  be  compared  with  expected 
routes,  positions  and  times  and  from  their  degree 
of  compliance  a  judgement  can  be  made  regarding 
their  allegiance. 

•  Data  Link  ID  -  with  a  secure  data  link  established 
between  friendly  platforms  and  the  NEMESIS 


demonstrator,  a  co-operative  target’s  allegiance 
may  be  assumed  with  a  high  degree  of  confidence. 
Four  of  the  above  systems  (The  IFF  system,  the  visual 
and  infra-red  imaging  devices  and  the  ESM  package) 
can  all  be  tasked  by  the  sensor  management  system  to 
steer  on  to  detected  targets  for  narrow  field  of  view 
active  or  passive  interrogation,  as  appropriate.  This 
requires  sophisticated  processing  (initially  a  major 
hurdle),  especially  in  the  case  of  fast  moving,  close  in 
targets,  where  a  few  seconds  delay  can  result  in  the 
target  never  being  in  the  field  of  view  of  the  sensor. 
The  data  fusion  block  diagram  is  shown  in  Figure  2. 


FIGURE  2:  THE  DATA  FUSION  BLOCK  DIAGRAM 


6.  THE  DATA  FUSION  PROCESS 

The  ID  CP  employs  a  Bayesian  algorithm  to  fuse  the 
ID  data  from  the  sensors  and  sources  if  they  have  been 
associated  with  a  track.  The  algorithm,  described  later 
in  this  document,  derives  probabilities  of  hypotheses 
given  an  observation  and  an  ‘a  priori’  estimation  of  the 
observation  being  possible.  The  Identification  Data 
Combining  Process  consists  of  two  major  sub¬ 
processes,  sensor  processing  and  fusion,  both 
described  in  the  paragraphs  below  and  depicted  in 
Figure  2. 

7.  SENSOR  PROCESSING 

Declarations  on  air  targets  from  disparate  sensors  can 
take  a  number  of  very  different  forms.  The  declaration 
can  range  from  an  aircraft  characteristic  matching  an 
entry  in  a  database  to  a  correct  response  from  an 
interrogation  from  the  ground.  _  Before  any 
identification  data  can  be  combined  or  fused  the  data 


has  to  take  a  common  format.  This  is  accomplished  in 
two  stages,  by  the  use  of  a  source  probability  matrix 
and  a  mapping  matrix.  The  first  stage  in  this  process 
takes  a  sensor  declaration  and  determines  a  level  of 
confidence  in  its  accuracy.  An  example  of  this  stage  is 
described  as  follows:  Consider  a  Mode  3/A  IFF  sensor 
and  its  possible  declarations  of  Evaluated  Correct 
Response  (ECR),  Evaluated  Incorrect  Response  (EIR), 
and  Evaluated  No  Response  (ENR).  The  sensor  would 
normally  interrogate  a  target  and  return  either  a  no 
response  or  a  four  digit  octal  code.  The  code  received 
is  compared  with  a  library  and  deemed  to  be  a  correct 
or  an  incorrect  response.  The  causes  for  a  particular 
response  however  are  not  conveyed  (e.g.  incorrect 
code  entered  in  the  transponder,  transponder  not 
working)  and  so  the  confidence  in  the  declaration  is 
lost.  The  source  probability  matrix  contains 
information,  derived  from  a  system  engineering 


32-4 


evaluation,  that  attaches  a  confidence  level  to  a  declaration  and  an  example  is  shown  in  Figure  3. 
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FIGURE  3  :  IFF  MODE  3/A  -  TYPICAL  SOURCE  PROBABILITY  MATRIX  (SPM) 


The  system  engineering  evaluation  needed  for  the 
population  of  these  matrices  is  considered  to  be  one  of 
the  more  difficult  tasks  in  implementing  the  IDCP  and 
it  should  be  noted  that  the  values  in  this  and  the 
following  figures  are  for  descriptive  purposes  only. 
The  matrix  allows  factors  such  as  terrain  screening, 
garbling  and  transponder  reliability  to  be  taken  into 


account,  building  in  a  level  of  declaration  probability. 
The  matrix  is  used  every  time  a  declaration  is  made  by 
the  sensor.  The  likelihood  vector  is  created  by  using 
the  SPM  row  corresponding  to  the  declaration  made. 
A  declaration  of  EIR  would  produce  a  likelihood 
vector  as  shown  in  Figure  4. 
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FIGURE  4  :  IFF  MODE  3/A  -  TYPICAL  SOURCE  DEPENDANT  LIKELIHOOD  VECTOR 


This  is  the  end  of  the  first  phase  of  data  formatting  but 
it  should  be  noted  that  the  likelihood  vector  is  still 
sensor  dependant.  The  second  stage  in  this  process 
converts  the  source  dependant  likelihood  vector  into  a 
common  format,  which  is  an  estimate  of  allegiance, 
(own  forces  (OF),  non-aligned  forces  (NA)  or  enemy 
forces  (EF)).  This  stage  involves  the  combination  of 
the  source  dependant  likelihood  vector  and  the 
mapping  matrix  by  matrix  multiplication.  The 


mapping  matrix  contains  more  ‘a  priori’  information 
expressing  probabilities  of  an  allegiance  making  a 
declaration,  e.g.  there  is  a  high  probability  that  own 
forces  will  make  correct  responses.  This  matrix  also 
accommodates  more  unexpected  events  such  as  the 
probability  of  friendly  aircraft  having  either  no  or 
defective  IFF  equipment.  A  typical  mapping  matrix  is 
shown  in  Figure  5. 
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FIGURE  5  :  IFF  MODE  3/A  -  TYPICAL  MAPPING  MATRIX  (MM) 


The  combination  of  the  source  dependant  likelihood 
vector  and  the  mapping  matrix  results  in  a  likelihood 
vector  which  is  no  longer  source  dependant  and 
instead  is  a  sources’  estimate  of  allegiance.  All  source 


types,  IFF,  ESM,  airspace  control,  for  example, 
produce  data  in  such  a  form  and  this  data  is  eligible  to 
be  fused.  An  example  of  an  allegiance  likelihood 
vector  is  shown  in  Figure  6. 
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FIGURE  6  :  TYPICAL  ALLEGIANCE  LIKELIHOOD  VECTOR 
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8.  Data  Fusion  within  the  IDCP 

The  previous  section  describes  the  generation  of  a 
likelihood  vector  from  a  sensor  declaration,  and  each 
source  type  produces  likelihood  vectors  in  this  format 
(‘Allegiance  LVs’  in  Figure  2).  This  section  describes 
the  method  of  fusing  data  from  the  individual  sensors. 


‘Fusion  Part  1’  in  Figure  2  combines  each  likelihood 
vector  from  the  different  source  types  to  produce  a 
single  likelihood  vector  for  the  target.  This  operation 
involves  multiplication  of  similar  matrices  to  produce 
the  single  vector.  An  example  of  such  a  target 
likelihood  vector  is  shown  in  Figure  7. 


OF 

NA 

EF 

0.60 

0.30 

0.10 

FIGURE  7  :  TARGET  LIKELIHOOD  VECTOR 


This  single  vector  is  passed  to  the  second  stage  ‘Fusion  a  normalised  ratio  of  OF:NA:EF.  An  example  of  the 

Part  2’  in  Figure  2,  which  incorporates  an  estimate  of  format  of  this  information  is  shown  in  Figure  8. 

the  force  mix  of  aircraft  in  the  local  area,  expressed  as 
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FIGURE  8  :  TYPICAL  FORCE  MIX  RATIO 


This  information  provides  the  prior  probabilities  required  for  Bayes  theorem  as  shown  below. 


Mow)-  p(®‘\ Oj) •  p(°j)  - 

'  ]  2]  P(Di\  Oj)  •  p(Oj) <_ 

_ Fusion  of  sensor 

and  a  priori  data 

—  Normalising 

all  j 

function 

where  : 

p(jOj\Di)  =  Final  ID  Likelihood  Vector 
p(^Di\ Oj')  =  Target  Likelihood  Vector 
pi^Oj)  =  Estimate  of  Force  Mix 

Bayes’  theorem  fuses  the  target  likelihood  vector 
produced  by  stage  one  of  fusion  and  the  ‘a  priori’ 
force  mix  ratio  (FMR)  to  form  a  final  normalised 


identification  likelihood  vector  for  the  target  being 
observed.  An  example  of  such  a  vector  is  shown  in 
Figure  9. 
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FIGURE  9  :  FINAL  ID  LIKELIHOOD  VECTOR 


Throughout  the  above  description  of  the  IDCP  process, 
the  sensor  declarations  being  used  are  assumed  to 
indicate  an  allegiance.  Sensors  and  sources  such  as 
IFF  and  ACP  supply  data  from  which  the  allegiance  of 
a  platform  can  be  directly  determined.  Some  sensors 
and  sources  however  supply  information  from  which 
allegiance  cannot  be  directly  determined.  ESM  and 
visual  identification  for  example  can  supply  platform 
information  only.  This  data  can  not  only  be 


operationally  valuable  in  its  own  right  but  can  also  be 
converted  into  an  estimate  of  allegiance  if  it  is  known 
which  nations  operate  certain  platforms.  Unfortunately 
with  many  nations  often  operating  the  same  platform 
type,  it  can  be  difficult  to  obtain  an  estimate  of 
allegiance  with  a  high  degree  of  confidence.  In 
addition  to  the  fusion  of  allegiance  information,  the 
IDCP  accommodates  this  fusion  of  platform 
information.  The  same  process  is  adopted  for  this 
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option  with  an  output  of  a  likelihood  vector,  estimating 
the  platform  type  of  the  target  being  observed. 
Whether  allegiance  or  platform  information  is  fused 
the  final  result  of  the  IDCP  may  not  be  in  the  correct 
form  for  display.  This  demonstrator  uses  the  STANAG 
1241  categories  (friend,  assumed  friend,  unknown, 
neutral,  suspect,  or  hostile)  for  ID  indication  on  the 
recognised  air  picture  (RAP)  and  in  order  to  arrive  at 
these  categories,  the  three  state  output  (OF,  EF,  NA) 
from  the  IDCP  must  be  thresholded. 

9.  CONFLICTS 

Identification  data  from  different  individual  sensors 
and  sources  on  the  same  track  can  often  be  in 
contradiction.  The  IDCP  has  the  ability  to  detect 
conflicting  sensor  information  by  testing  the 
identification  data  at  each  combination  stage. 
Conflicting  identification  data  can  arise  for  a  number 
of  reasons;  mis-association  of  identification  data  to 
tracks  being  the  most  common.  A  malfunctioning 
sensor  can  also  cause  conflicting  information  and  it  is 
considered  that  if  a  conflict  occurs  the  identification 
officer  will  resolve  it  manually  by  either  breaking 
associations,  inhibiting  a  sensor  input  or  resetting  the 
fusion  process.  There  are  currently  no  means  by 
which  identification  data  conflicts  in  the  IDCP  can  be 
resolved  automatically. 

10.  TRIALS  PROGRAMME  AND  OPERATING 
PROCEDURES 

NEMESIS  was  deployed  to  two  operational  UK 
airfields  to  gather  and  analyse  live  data.  Targets  of 
opportunity  were  designated  friend,  foe,  and  neutral  to 
permit  assessment  of  system  performance.  Air  truth 
was  derived  from  numerous  sources  but  principally  the 
air  traffic  control  radar  picture  allied  with  IFF  code 
and  call  sign  data.  The  identification  of  targets  with 
NEMESIS  is  a  process  initiated  by  the  acquisition  of 
radar  tracks,  at  a  range  of  20-25  miles,  and  often 
confirmed  (in  the  case  of  friends)  with  an  IFF 
identification  and  an  IFF  derived  track.  Radar  and  IFF 
tracks  are  then  correlated  to  provide  the  global  track 
table  to  the  fusion  processor.  The  global  track  table 
prioritises  these  tracks  in  threat  priority  order,  based 
upon  range,  speed  and  direction  and  commands  the 
visual  and  infra-red  sensors  onto  the  correct  azimuth 
for  more  detailed  search.  Lack  of  any  target  height 
information  demands  that  the  ‘pointing’  sensors  then 
search  in  elevation  upon  that  bearing.  When  a  target 
comes  within  range  of  the  optical  and  infra-red 
sensors,  an  auto-track  routine  takes  over  (with  greater 
bearing  accuracy  and  less  latency  than  the  radar)  and 
the  platform  identification  is  performed  by  a  man-in- 
the-loop  who  inputs  data  via  a  menu  driven  touch 
screen.  In  parallel  with  this  reactive  process,  the  ACP 
identification  source  compares  the  position,  time  and 
direction  of  the  detected  targets  and  makes  a  prediction 
of  their  intention  to  follow  known  pre-arranged 


procedural  practices.  During  the  whole  phase  the  IFF 
continues  to  report  on  the  track  and  the  ESM  and 
acoustic  sensors  continue  to  ‘listen’,  and  association 
routines  attempt  to  tie  any  detected  ‘strobes’  to  known 
tracks.  The  link  sensor  also  passes  allegiance 
indications  forward  to  the  fusion  processor.  While 
these  automatic  processes  continue,  the  fusion  post 
operator  monitors  the  evolving  recognised  air  picture 
and  assesses  the  allegiance  recommendations  made  by 
the  Bayesian  fusion  engine.  Lack  of  ID  information  or 
conflicting  data  can  sometimes  be  remedied  by  a 
command  from  the  operator  to  the  IFF,  ESM,  visual 
and  thermal  sensors  to  make  repeated  ‘hits’  upon  the 
track  to  strengthen  knowledge  about  the  target. 
Graphical  and  tabular  fusion  displays  meanwhile 
describe  the  evolving  synergy  of  the  ID  picture.  The 
output  from  the  fusion  process  is  an  automatic 
classification  of  all  targets  in  cover,  updated  every  few 
seconds  with  a  recommended  STANAG  1241  target 
classification.  These  classifications  derive  from  the 
friend,  foe,  neutral  vectors  output  from  the  fusion 
process  after  thresholding.  Thresholds  are  set 
reflecting  any  current  state  of  tension. 

11.  DIFFICULTIES  ENCOUNTERED 

Two  primary  problems  were  encountered  during  the 
development  and  study  phases  of  the  demonstrator. 
The  primary  radar  regularly  formed  false  tracks  and 
failed  to  maintain  fast  manoeuvring  target  tracks 
resulting  in  missed  and  erroneous  ID  associations.  The 
second  problem  arose  with  high  traffic  densities  when 
the  processing  times  increased  with  the  added 
workload  causing  excessive  delays  between  the 
appearance  of  the  target  and  the  ID  declaration  stage. 
Additionally  the  characterisation  of  sensors,  the 
development  of  matrices  and  establishing  other  ‘a 
priori’  data  were  non- trivial  tasks  and  with  hindsight, 
attendance  at  a  full  scale  exercise  would  have  provided 
more  realistic  ‘enemy’  targets,  but  logistics  prevented 
this  from  happening. 

12.  CONCLUSIONS 

The  intention  to  practically  implement  the  STANAG 
4162  data  fusion  algorithm  was  satisfactorily  met  and 
the  risks  associated  with  the  use  of  this  form  of 
identification  data  fusion  are  now  better  understood. 
Study  of  the  quality  of  ID  accuracy  indicates  that  a  few 
high  quality  sensors  provide  more  accurate  results  than 
many  low  quality  sensors.  A  sophisticated  fusion 
algorithm  obviously  does  little  to  improve  the 
declarations  made  by  poor  quality  sensors.  Conflicts  of 
identification  that  occur  on  any  track  when  data  from 
two  sources  differ  are  flagged  to  the  operator  as  an 
automatic  by-product  of  fusion.  This  is  seen  as  an 
important  benefit  in  alerting  the  identification  officer 
who  is  able  to  intervene  and  correct  any  anomalies  that 
occur.  This  function  provides  early  alerting  to  an 
operator  when  a  source  or  sensor  becomes  unreliable. 
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The  major  obvious  benefit  is  that  of  the  effect  upon 
operator  workload.  An  in-service  system  scaled  to 
handle  greater  than  200  tracks  and  the  equivalent  of  5 
sensor  or  source  declarations  every  10  seconds  on  each 
track  for  example,  is  a  very  demanding  recognised  air 
picture  for  any  operator  to  assess  in  a  timely  fashion. 
This  automatic  classification  technique  enables  the  ID 
operator  to  manage  his  entire  area  of  responsibility  in  a 
prompt  and  timely  fashion  and  permits  him  more  time 
to  investigate  thoroughly  the  tracks  that  are  of  special 
interest,  (primarily  the  hostiles  and  unknowns). 

13.  FUTURE  PROGRAMME 

The  NEMESIS  programme  to  develop  a  STANAG 
4162  real-time,  single  node,  technical  demonstrator  is 
nominally  complete  and  all  the  investigations  are 
finished.  The  follow-on  stage,  which  is  in  its  infancy, 


will  assess  the  implications  and  problems  associated 
with  networking  up  to  four  NEMESIS  style  nodes  in  a 
flexible  architecture  with  simulated  military  data  links. 
The  programme  will  be  laboratory  based  unlike  the 
NEMESIS  single  node  and  will  draw  much  of  its  data 
from  the  live  environment.  STANAG  4162 
architecture  studies  are  progressing  and  the 
performance  of  centralised,  de-centralised  and 
autonomous  structures  will  be  addressed  during  this 
programme. 
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COMMENT/QUESTION: 

The  NC3A  also  conducted  an  IDCP  trial  with  live  data  based  on  information  recorded  during 
Central  Enterprise  NATO  exercise.  The  current  STANAG  4162  states  rules  for  war  time,  but  it  is 
very  difficult  to  work  with  in  peace/crisis  time  (typical  exercise  ....)  where  Modes  2,3  and  4  could 
be  used  simultaneously,  and  not  all  nations  use  them  with  the  same  rules. 

Question:  Did  you  find  any  problems  of  this  kind  during  the  NEMESIS  experiment? 


AUTHOR/PRESENTER’S  REPLY: 

This  problem  was  not  observed  as,  although  NEMESIS  was  capable  of  Mode  1,2,3  and  4 
interrogations,  only  Mode  3  replies  were  ever  received. 

NEMESIS  implemented  an  early  version  of  STANAG  4162  which  did  not  appreciate  peacetime 
IFF  problems.  This  part  of  the  STANAG  is  currently  being  reviewed  and  the  new  version  will 
specifically  address  peacetime  IFF  use. 
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Abstract 

Algorithms  for  managing  bandwidth-limited  communi¬ 
cations  links  in  a  multi-platform  and  multi-target  bat¬ 
tlespace  data  fusion  scenario  are  investigated.  The 
platform  architecture  is  modelled  on  a  fully  connected 
decentralised  system  based  on  a  simplified  tactical  air 
warfare  scenario.  Information  filters  on  each  plat¬ 
form  maintain  track  estimates  for  each  target.  These 
are  transmitted  around  the  system  in  accord  with  a 
time-division-multiple-access  communications  proto¬ 
col  based  on  JTIDS.  Here  we  address  a  practical  limi¬ 
tation  on  the  communications  bandwidth:  it  is  unable 
to  support  the  full  set  of  state  informationfor  all  of  the 
targets  in  each  communications  cycle.  Strategies  must 
therefore  be  found  to  manage  the  bandwith.  In  this 
paper  we  consider  two  different  communications  man¬ 
agement  algorithms:  (i)  a  somewhat  ad  hoc  cyclic, 
or  round-robin,  algorithm,  and  (ii)  a  more  principled 
information  theoretic  algorithm.  In  general  we  find 
that  the  system,  has  better  tracking  performance  under 
the  information  theoretic  algorithm  than  it  does  for 
the  round-robin  algorithm.  This  result  appears  to  be 
fairly  robust  to  uncertainties  in  a  transmitting  plat¬ 
form’s  prediction  of  the  state  estimates  that  are  cur¬ 
rently  being  maintained  at  the  recipient  platforms.  In 
practice  -  since  the  information  theoretic  algorithm 
is  more  computationally  costly  to  implement  than  the 
round-robin  algorithm  -  the  decision  of  whether  or  not 
to  employ  intelligent  communications  management  is 
likely  to  depend  upon  the  severity  of  the  bandwidth 
constraint,  the  number  and  type  of  tracked  targets,  and 
the  predictive  power  of  a  platform  with  regard  to  other 
platforms  in  the  network. 

1.  Introduction 

Multisensor  data  fusion  is  expected  to  play  a  crucial 
role  in  the  digital  battlespace  of  the  21st  century  [1]. 
Arguably  one  of  the  greatest  challenges  to  be  faced  is 
the  increasingly  complex  nature  of  battlespace  scenar¬ 
ios,  and  the  huge  volumes  of  disparate  data  that  these 
will  generate.  This  may  conceivably  strain  the  sens¬ 
ing  resources,  computer  processing  speed  and  mem¬ 
ory,  and  communications  bandwidth  to  which  a  sys¬ 
tem  has  reasonable  access.  Algorithms  for  resource 
management  are  therefore  likely  to  be  an  important 
element  of  future  battlespace  data  fusion  systems. 


A  number  of  authors  have  investigated  system  re¬ 
source  management  in  multisensor  networks.  How¬ 
ever,  this  effort  has  almost  entirely  been  focussed  on 
sensor  management  [2,  3,  4],  It  has  led  to  a  principled 
information  theoretic  approach  to  sensing  resource  al¬ 
location.  In  contrast,  there  has  been  relatively  little 
published  work  that,  provides  a  systematic  investiga¬ 
tion  into  the  issues  that  are  involved  in  managing  a 
limited  communications  bandwidth  resource  in  mul¬ 
tisensor  systems.  This  is  somewhat  surprising  given 
that  communications  issues  may  be  critical  in  mili¬ 
tary  applications.  Aircraft  on  an  intercept  mission 
may,  for  example,  wish  to  limit  their  usage  of  commu¬ 
nications  bandwidth  in  order  to  achieve  some  measure 
of  stealth.  Alternatively,  reduced  bandwidth  can  arise 
as  a  result  of  signal  jamming  by  the  enemy  aircraft. 

Our  previous  work  on  resource  management  in  multi¬ 
sensor  architectures  has  considered  the  relative  perfor¬ 
mance  of  two  different  approaches  to  communications 
management  in  decentralised  identity  fusion  [5]  and 
tracking  [6]  systems.  One  approach  was  ad  hoc  but 
computationally  cheap,  the  other  approach  was  firmly 
rooted  in  information  theory  but  involved  more  calcu¬ 
lation.  Process  models  were  developed  to  indicate  the 
conditions  under  which  the  information  theoretic  ap¬ 
proach  was  likely  to  be  more  desirable  than  the  simple 
alternative.  This  current  work  is  intended  to  revisit 
the  process  model  for  the  decentralised  tracking  sys¬ 
tem,  in  the  light  of  more  realistic  simulations  that 
permit  us  to  explore  the  robustness  of  the  informa¬ 
tion  theoretic  approach  to  modelling  inaccuracies. 

The  paper  is  organised  as  follows.  Section  2  covers 
some  of  the  relevant  background  in  multisensor  sys¬ 
tems  architectures,  decentralised  filtering,  and  infor¬ 
mation  theory.  Section  3  outlines  a  computer  simu¬ 
lation  of  a  simple  decentralised  tracking  system.  Sec¬ 
tion  4  describes  the  communications  management  al¬ 
gorithms  in  some  detail.  The  simulated  battlespace 
scenario  and  numerical  performance  diagnostics  are 
described  in  Section  5.  Section  6  presents  our  main 
results  and  these  are  discussed  in  Section  7.  Finally, 
Section  8  concludes  and  summarises  the  paper. 

2.  Background 

This  section  presents  some  brief  background  material 
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on  the  key  concepts  -  decentralised  systems,  state  es¬ 
timation,  and  information  theory  -  that  underpin  the 
work  described  here.  Further  details  can  be  found  in 
the  cited  references. 

2.1  Decentralised  Systems 

Centralised  multisensor  systems  are  characterised  by 
having  a  central  processing  node  through  which  all  of 
the  data  from  around  the  distributed  sensing  network 
is  channeled.  This  architecture  can  result  in  catas¬ 
trophic  failure  of  an  aircraft  intercept  mission,  for 
example,  if  the  inter-platform  communications  band¬ 
width  is  reduced  or  lost  -  a  highly  probable  outcome 
in  battlespace  scenarios.  Successive  levels  of  decen¬ 
tralisation  add  some  measure  of  fault  tolerance  to  a 
multisensor  system.  The  extreme  case  is  a  fully  de¬ 
centralised  architecture  with  no  central  facility  of  any 
kind.  This  is  the  architecture  that  we  have  investi¬ 
gated  in  our  previous  work  [5,  6],  and  is  also  the  focus 
of  this  present  investigation.  In  further  references  to 
decentralised  systems  in  this  paper,  full  decentralisa¬ 
tion  with  full  network  connectivity  should  be  assumed. 
Fig.l  illustrates  a  range  of  different  multisensor  sys¬ 
tem  architectures. 
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ity  criteria  under  certain  conditions  [8].  A  variant  of 
the  Kalman  filter,  known  as  the  information  filter,  is 
utilised  here.  This  is  expressed  in  terms  of  the  infor¬ 
mation  state  variables,  y  and  Y,  that  are  related  to 
the  standard  Kalman  state-space  variables,  x  and  P, 
as  follows: 

Y(*|i)  =  P_1(*li);  y(*li)  =  P-1(*U)x(i|j)  (l) 

where  («|j)  refers  to  an  estimate  at  time  step  i  based 
on  all  the  observations  up  to  time  step  j  (where  j  <  i). 
The  information  filtering  equations  are  straightfor¬ 
ward  to  decentralise,  with  local  measurement  update 
and  global  data  assimilation  terms  being  much  less 
complex  than  those  for  the  Kalman  filter  [7]. 

2.3  Information  Theory 

The  information  content,  7,  of  a  multivariate  proba¬ 
bility  density  function,  /(x),  is  defined  as: 

7=<ln/(x)}  (2) 

where  (•)  denotes  statistical  expectation.  In  tracking 
applications,  x  refers  to  the  state  vector  of  a  target 
and  the  uncertainty  in  the  components  of  x  is  usu¬ 
ally  described  by  a  multivariate  Gaussian  distribution. 
This  can  be  expressed  as  follows: 

/(x;x,P)=  (27r)l/!|p|i/2e*P  -^(x-x)P_1(x-x)T 

(3) 

where  x  is  the  mean  value  of  the  state  vector,  P  is  the 
state  error  covariance  matrix  (|P|  is  the  determinant 
of  P),  and  l  is  the  dimension  of  the  state  vector.  The 
information  relating  to  a  target’s  track  is  then  given 
by: 

7t  = -iln[(27re)J|p|]  (4) 

or  equivalently  by: 


Figure  1:  Multisensor  System  Architectures 


I*  =  iln[(2*e)-'|Y|]  (5) 


In  addition  to  being  resilient,  decentralised  architec¬ 
tures  offer  a  number  of  other  features  that  are  desir¬ 
able  for  battlespace  systems.  These  include  modu¬ 
larity,  scalability,  and  flexibility  [7].  It  can  be  shown 
that  decentralised  systems  require  less  communication 
bandwidth  than  their  centralised  counterparts  to  ob¬ 
tain  identical  fused  identity  or  track  estimates.  This 
advantage  is  offset  by  an  increase  in  the  number  of 
communications  links  required. 

2.2  State  Estimation 

The  most  common  state  estimator  is  the  Kalman  fil¬ 
ter,  which  is  proven  to  satisfy  a  number  of  optimal- 


where  t  is  the  identity  index  of  the  target.  In  fact  later 
we  shall  only  be  interested  in  comparing  relative  in¬ 
formation  values  between  the  targets  state  estimates 
and  not  in  the  absolute  information  values  for  indi¬ 
vidual  targets  track  estimates.  This  means  that  the 
constant  term  in  the  expansion  of  the  logarithm  drops 
out.  Also,  since  the  logarithm  of  |Y|  is  monotonically 
increasing  with  |Y|  we  are  able  to  replace  In  |Y|  sim¬ 
ply  with  | Y  | .  Therefore  let  us  define  the  comparative 
information  value  to  be: 

I't  =  |Y 


(6) 
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Hence,  the  track  information  at  a  given  node  (i.e.  a 
sensor  and  its  processer)  can  by  defined  as: 

(7) 

vt 

where  n  is  the  index  of  the  node.  Similarly,  the  system 
information,  Is ,  is  given  by: 

7S=X>  (8) 

Vn 

where  S  indicates  system  information. 

3.  Simulation  Details 

A  computer  simulation  was  developed  to  investi¬ 
gate  communications  management  in  a  simple  decen¬ 
tralised  tracking  system.  This  paper  only  addresses 
the  effect  that  communications  management  has  upon 
one  aspect  of  low-level  data  fusion,  namely  the  track¬ 
ing  of  multiple  targets  from  multiple  platforms.  In 
this  section  the  components  of  the  simulator  are  de¬ 
scribed.  Our  generic  model  of  a  decentralised  system 
-  developed  in  conjunction  with  Hugh  Durrant- Whyte 
of  the  University  of  Sydney  -  is  illustrated  in  Fig.  2. 
Although  situation  and  threat  assessment  are  shown 
in  this  diagram,  their  inclusion  in  the  simulator  is  de¬ 
ferred  to  a  future  investigation.  A  further  sophistica¬ 
tion  that  we  have  also  chosen  not  to  include  in  the 
simulator  at  this  stage  is  a  feedback  loop  that  allows 
targets  to  respond  reactively  to  the  behaviour  of  the 
platforms,  see  A  in  Fig.  2.  In  addition,  the  communi¬ 
cations  model  is  ideal  in  the  sense  that  all  transmitted 
data  are  received  successfully,  see  B  in  Fig.  2. 

3.1  World  Generator 

The  world  is  populated  with  Nt  targets  of  three  basic 
types.  One  type  is  meant  to  be  representative  of  a 
bomber  aircraft,  another  is  representative  of  a  fighter 
aircraft,  and  the  third  type  is  representative  of  a  high 
performance  (advanced)  fighter  aircraft.  All  of  the 
targets  execute  a  single  turning  manoeuvre  during  the 
simulation  interval.  The  bombers  turn  with  a  maxi¬ 
mum  acceleration  of  B g,  the  fighters  at  Fg,  and  the 
advanced  fighter  at  A g.  Here,  A  >  F  >  B  which  infers 
that  the  advanced  fighter  is  the  most  and  the  bomber 
the  least  manoeuvrable  targets. 

3.2  Platforms 

The  world  is  observed  from  three  platforms  which 
may,  in  a  tactical  air  warfare  scenario,  represent  an 
Airborne  Warning  and  Control  System  (AWACS)  air¬ 
craft  and  a  lead/wingman  pair  of  fighter  aircraft.  The 
platforms  form  a  fully  connected  system  and  it  is  as¬ 
sumed  that  the  targets  are  visible  from  all  three  plat¬ 
forms  at  all  times. 


. _ _ Platform  , 

( Threat  Assessment  ] 

(  Situation  Assessment  J 

- 

-►(Sensors  )*-*{ Estimation  )^*^(Communications  V--j* 

To  other 
Platforms 

The 

World 

W 

o 

(d 

(Threat  Assessment  j 

c 

3 

E  aj 

E  tj 

(  Situation  Assessment 

o  o 

O  2 

- 

-►(Sensors  V -►(Estimation  )•*-■{ Communications  V— !► 

To  other 
Platforms 

Figure  2:  Components  in  a  Decentralised  Tracking 
System 

3.3  Sensors 

Each  platform  is  equipped  with  a  single  sensor.  The 
sensors  measure  the  x  and  y  range  of  the  targets  in  2d 
Cartesian  space.  These  measurements  are  synchro¬ 
nised  at  the  platforms  and  are  updated  at  constant 
time  intervals.  All  of  the  sensors  have  a  fixed  observa¬ 
tion  noise  covariance  R,.  The  noise  in  the  individual 
sensors  is  independent  in  the  x  and  y  range  measure¬ 
ments.  The  noise  is  also  independent  between  the  dif¬ 
ferent  sensors. 

An  important  feature  of  these  simulations  is  that  we 
have  fixed  the  probability  of  target  detection,  Pd,  at 
the  sensors  to  a  value  that  is  less  than  unity.  This 
introduces  some  uncertainty  into  a  transmitting  plat¬ 
forms  prediction  about  the  track  state  error  covari¬ 
ances  that  are  being  maintained  at  the  recipient  plat¬ 
forms.  The  reason  is  that  at  any  given  time  a  plat¬ 
form  will  not  know  whether  the  other  platforms  have 
based  their  current  state  estimate  for  a  target’s  track 
on  a  new  observation,  or,  in  the  absence  of  detecting  a 
target,  simply  predicted  forward  the  previous  state  es¬ 
timate  for  the  corresponding  track.  The  implications 
of  this  for  communications  management  are  discussed 
later. 

3.4  Estimation 

The  targets  are  tracked  with  information  filters. 
These  assume  a  constant  velocity  process  model  for 
the  targets  with  a  process  noise  variance  tuned  to  their 
accelerations  [9].  Local  state  estimation  is  performed 
separately  at  each  of  the  platforms  and  the  local  esti¬ 
mates  are  fused  into  global  estimates  following  a  data 
assimilation  step  after  each  communication  cycle.  In 
order  to  avoid  tricky  data  association  problems,  which 
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are  not  really  the  issue  here,  measurements  are  cor¬ 
rectly  assigned  to  tracks  from  within  the  simulation 
software.  Local  and  global  estimation  does  not  there¬ 
fore  suffer  from  incorrect  data  association  here,  al¬ 
though  this  would  be  an  issue  in  a  real  system. 

3.5  Communications 

The  communications  protocol  that  is  followed  in  the 
simulator  is  a  time-division-multiple-access  system, 
based  on  JTIDS  [10].  All  the  communication  links  are 
assumed  to  be  maintained  throughout  the  simulation 
and  to  have  no  latency.  Each  platform  in  the  network 
is  designated  a  time-slot  in  which  it  sends  a  burst  of 
formatted  data.  Here  we  consider  the  data  to  com¬ 
prise  of  state  estimates  and  associated  error  covari¬ 
ances  for  up  to  Nt  targets,  depending  on  the  severity 
of  the  bandwidth  constraint.  In  decentralised  systems 
one  must  be  careful  to  handle  the  problem  of  mul¬ 
tiple  data  counts.  These  can  lead  to  over  estimated 
filters,  which  are  clearly  undesirable  in  mission  critical 
systems.  In  fully  connected  decentralised  systems  the 
problem  is  solved  by  maintaining  an  additional  filter, 
known  as  the  channel  filter  [11].  The  communications 
module  in  our  software  is  responsible  for  updating  the 
channel  filters  and  implementing  the  communications 
management  algorithms.  The  latter  component  of  the 
module  is  described  in  more  detail  in  Section  4. 

3.6  Implementation  Details 

The  simulator  has  been  coded  in  the  ‘C’  program¬ 
ming  language  with  results  displayed  from  the  Tcl/Tk 
scripting  language.  Simulations  are  carried-out  by 
running  the  code  on  a  Sun  Ultra  140E  networked  com¬ 
puter  under  the  Unix  operating  system.  Results  are 
stored  to  file  for  off-line  analysis. 

The  two  communications  management  algorithms  are 
run  concurrently  during  the  simulation.  This  presents 
the  same  data  set  to  each  algorithm  during  a  par¬ 
ticular  simulation.  Hence,  a  reasonable  comparison  of 
performance  can  be  carried-out  on  a  single  simulation. 
This  is  useful  for  demonstration  purposes. 

4.  Communications  Management 

We  now  turn  to  the  main  feature  of  our  investigation: 
the  algorithms  for  managing  limited  communications 
resources  in  a  decentralised  tracking  system.  Here  the 
following  somewhat  arbitrary  restriction  is  placed  on 
the  communications  bandwidth:  it  is  sufficient  to  sup¬ 
port  the  transmission  of  state  information  (estimation 
and  covariance)  for  only  Nj,  out  of  Nt  targets  in  each 
communications  cycle.  The  management  problem  is 
basically  to  decide  which  are  the  most  appropriate  tar¬ 
gets  on  which  to  communicate  track  data.  We  have 
taken  two  approaches  to  this  problem.  These  are  de¬ 
scribed  below. 


4.1  Round-Robin  Algorithm 

This  is  a  computationally  cheap  and  ad  hoc  algorithm. 
The  transmitting  platform  essentially  selects  the  tar¬ 
gets  on  which  to  communicate  state  information  in 
a  wraparound  cyclic,  or  round-robin,  manner.  The 
round-robin  algorithm  is  initalised  randomly  on  each 
platform  by  shuffling  the  Nt  indices  that  label  the 
targets  and  then  transmitting  data  on  the  targets  that 
correspond  to  the  first  Nf  indices  in  the  list.  Consider 
a  scenario  in  which  Nt  =  5  and  IV 'T  —  3.  Suppose  the 
initialisation  process  at  one  of  the  platforms  selects 
targets  T\,Ts,  and  T5  for  the  first  communication  cy¬ 
cle.  The  next  time  that  this  platform  communicates 
it  will  select  targets  T2,T4,  and  T).  This  is  followed 
by  targets  T3,  T5,T2,  and  so  on  until  the  simulation  is 
complete. 

4.2  The  Information  Theoretic  Algorithm 

This  is  a  principled  approach  to  communications  man¬ 
agement  based  on  information  theory.  It  is  computa¬ 
tionally  more  expensive  than  the  round-robin  algo¬ 
rithm  to  implement.  The  fundamental  idea  is  that  a 
transmitting  node  communicates  data  corresponding 
to  the  N!p  targets  that  will  produce  the  maximal  over¬ 
all  change  in  information,  in  a  single  communications 
cycle,  at  the  receiving  platforms. 

The  mathematical  notation  for  this  communications 
management  algorithm  is  somewhat  involved.  Hence, 
for  clarity  and  simplicity  we  provide  a  worded  descrip¬ 
tion  of  its  behaviour.  The  algorithm  begins  with  the 
transmitting  node  estimating  the  total  system  infor¬ 
mation  before  a  communication  has  been  made.  This 
results  in  a  single  information  value  being  generated 
for  each  iteration  of  the  algorithm.  This  is  referred  to 
as  the  prior  system  information  ( estimate )  given  by: 

prior  prior 

system  transmitter  receivers 

information  =  information  +  information 

( estimate )  (real)  ( predicted ) 

(9) 

In  the  second  stage  of  the  algorithm  the  transmitting 
node  estimates  the  total  system  information  after  a 
communication  has  been  made  on  each  target,  i.e.  Nt 
values  in  all.  This  is  referred  to  as  the  posterior  system 
information  (estimate)  and  is  given  by: 


posterior  posterior 

system  transmitter  receivers 

information  —  information  +  information 

( estimate )  (real)  (predicted) 

(10) 

The  third  stage  of  the  algorithm  is  to  produce  a  list, 
comprising  Nt  elements,  of  the  change  in  information 
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value  that  will  occur  on  transmission  of  data  relating 
to  a  particular  target.  These  are  referred  to  as  decision 
values  and  are  calculated  as: 


posterior 

prior 

decision 

system  — 

system 

value  = 

information 

information 

( estimate ) 

( estimate ) 

The  final  stage  of  this  algorithm  is  to  sort  the  decision 
values  into  descending  order.  Data  on  the  first  Nj. 
indexed  targets  in  the  list  are  then  transmitted  to  the 
other  nodes. 

This  information  theoretic  algorithm  implies  that  the 
transmitting  platform  has  knowledge  of  the  informa¬ 
tion  state  at  the  receiving  platforms.  In  our  previous 
work  [5]  this  was  the  case,  effectively  since  the  plat¬ 
forms  and  sensors  were  identical  and  Pd  =  1.  Clearly 
this  was  a  somewhat  unrealistic  assumption  and  we 
have  relaxed  it  artifically  by  introducing  noise  to  the 
decision  values  before  they  are  sorted  into  descending 
order.  This  permits  us  to  investigate  the  robustness 
of  the  information  theoretic  communications  manage¬ 
ment  algorithm  to  inaccuracies  in  the  decision  process. 
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Figure  3:  The  Multi-Platform  and  Multi-Target 
Tracking  Scenario 


5.  Investigation  Process 

This  section  of  the  paper  describes  the  battlespace 
scenario,  inter-platform  communications  scenario,  and 
the  statistical  analysis  methods  that  were  applied  to 
the  simulation  results. 

5.1  Battlespace  Scenario 

The  scenario  on  which  the  computer  simulations  were 
executed  is  illustrated  in  Fig.  3.  We  have  somewhat 
arbitrarily  chosen  values  of  Nt  =  5  targets  (shown  as 
black  aircraft  in  the  diagram)  and  a  decentralised  sys¬ 
tem  comprising  3  platforms  (white  aircraft).  This  sce¬ 
nario  is  modelled  on  a  simplistic  air  warfare  scenario 
in  which  ‘possible  hostile’  aircraft  are  intercepted  by 
‘friendly’  aircraft  comprising  of  a  lead/wingman  pair 
in  communication  contact  with  an  Airborne  Warning 
And  Control  System  (AWACS)  aircraft.  The  scenario 
is  not  intended  to  reflect  a  true  ‘intercept’  mission. 
However,  a  number  of  realistic  elements  are  evident, 
these  include:  multi-aircraft  friendly  force,  multi¬ 
aircraft  hostile  force,  imperfect  sensors,  and  band¬ 
width  constrained  data  communication  channels. 

The  ‘possible  hostile’  aircraft  comprise  two  large 
bomber  aircraft  performing  turning  manoeuvres  with 
an  acceleration  of  B g.  These  aircraft  are  flanked  by 
two  fighter  aircraft  and  one  advanced  fighter  aircraft 
that  are  able  to  perform  higher  turning  manoeuvres 
of  F<jr  and  kg  respectively.  All  the  aircraft  fly  with 
slightly  different  speeds.  Their  mission  begins  with 


all  the  aircraft  flying  in  parallel.  However,  on  detect¬ 
ing  that  they  are  being  observed,  the  fighter  aircraft 
perform  maximum  acceleration  manoevers  for  turning 
angles  of  approximately  +60°,  —90°  and  +45°.  Dur¬ 
ing  this  time  the  bomber  aircraft  maintain  their  origi¬ 
nal  paths.  The  trajectories  for  all  the  ‘possible  hostile’ 
aircraft  are  represented  (approximately)  by  the  asso¬ 
ciated  arrowed  lines  in  Fig.  3. 

The  ‘friendly’  aircraft  maintain  parallel  flight  trajec¬ 
tories  through  the  portion  of  the  intecept  mission  that 
is  simulated  here.  The  lead/wingman  pair  fly  ahead  of 
the  AWACS  and  towards  the  ‘possible  hostile’  aircraft. 
During  the  mission  it  is  assumed  that  all  of  the  ‘possi¬ 
ble  hostile’  aircraft  are  in  the  field  of  views  (FOVs)  of 
all  the  ‘friendly’  aircraft  all  of  the  time.  Further,  it  is 
assumed  that  the  sensor  information  generated  by  all 
three  ‘friendly’  aircraft  are  comparable.  For  simplicity 
of  simulation  the  assumption  is  made  that  sensor  clut¬ 
ter  levels  and  distance  between  the  ‘possible  hostile’ 
aircraft  are  such  that  near  perfect  data  association 
is  maintained.  This  also  implies  that  the  number  of 
spurious  tracks  that  are  generated  is  negligible. 

5.2  Communications  Scenario 

For  the  work  presented  in  this  paper  we  are  concerned 
with  a  performance  comparison  of  the  communica¬ 
tions  management  algorithms  as  the  precision  of  the 
decision  values  generated  by  the  information  theoretic 
approach  is  varied.  This  is  achieved  by  introducing  a 


33-6 


percentage  error  in  the  decision  values  before  deciding 
which  data  to  transmit.  Arbitrary  values  for  these  er¬ 
rors  of  of  10,  50  and  90  %  were  selected.  These  errors 
are  introduced  as  follows.  First,  a  sum  of  the  decision 
values  is  calculated.  This  allows  the  percent  error  in 
decision  values  to  be  represented  as  an  information 
error  in  decision  values,  e.g.  10%  of  the  sum  of  the 
decision  values.  An  equal  portion  of  this  information 
error  is  then  subtracted  from  each  decision  value.  The 
information  error  in  decision  values  is  then  randomly 
redistributed  over  the  decision  values.  For  each  value 
of  percent  error  in  the  decision  values,  simulations  are 
performed  with  the  bandwidth  constrained  to  be  a 
percentage  of  the  full  bandwidth  that  is  required,  i.e. 
N^/Nt  x  100,  as  N/p  is  varied  from  0  to  5.  This  gives 
values  of  0,  20,  40,  60,  80  and  100%. 


5.3  Analysis 


Although  the  tracking  and  communications  manage¬ 
ment  algorithms  are  employed  in  information  space, 
it  is  more  meaningful  to  produce  the  results  in  stan¬ 
dard  space  state.  This  does  not  affect  the  advantages 
of  implementing  the  information  form  of  the  Kalman 
filter  as  discussed  in  Section  2.2,  since  the  analysis 
is  performed  off-line.  Hence,  the  performance  of  the 
two  communications  management  algorithms  are  con¬ 
sidered  in  the  context  of  the  determinant  of  the  co- 
variances  produced  by  the  tracking  algorithm,  P.  In 
general,  this  quantity  provides  a  measure  of  the  vol¬ 
ume  of  the  uncertainty  ellipsoid  associated  with  the 
track  estimate. 


The  results  are  presented  from  the  view  point  that  the 
‘friendly’  aircraft  have  a  system  requirement  to  track 
the  targets  with  a  covariance  that  should,  on  aver¬ 
age,  not  exceed  Preg.  This  value  is  set  during  system 
specification.  Hence,  the  results  are  presented  using 
the  following  average  (ave)  and  standard  deviation  (<r) 
values  for  each  target: 
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where: 


j  P  j  -  Is  the  determinant  of  the  state 
error  covariance  on  a  track 
xx  -  Denotes  the  round-robin  (RR)  or 

information  based  (IN)  communications 
management  algorithm. 

Nm  -  The  number  of  Monte  Carlo  simulations 
carried-out,  i.e.  10  for  the  work 
presented  in  this  paper, 
m  -  The  index  of  a  particular  simulation 
i.e.  m  £  {1,2, .  ..,NM }• 

Nk  -  The  number  of  discrete  time  intervals 
in  a  simulation  (after  the  track  has 
been  initialised),  i.e.  55  for  the 
work  presented  here. 

k  -  The  index  of  a  particular  discrete 

time  interval,  i.e.  k  £  { 1,2,...,  Nk}- 
Nn  -  The  number  of  nodes  or  platforms  used 
in  the  simulation,  i.e.  3  for  the  work 
presented  in  this  paper, 
n  -  The  index  of  a  particular  node  or 
platform,  i.e.  n  £  {1, 2, . . . ,  N^}- 
Ns  -  Number  of  sample  points,  where 
Ns  =  Nm  +  Nk  +  N. 

These  average  and  standard  deviation  values  are  used 
to  statistically  summarise  the  results  of  this  investiga¬ 
tion. 

6.  Results 

The  main  results  are  presented  in  this  section.  Fig.  4 
provides  results  for  three  different  percent  error  in 
decision  values  of  10,  50  and  90  %  in  plots  (a),  (b) 
and  (c)  respectively.  Here  the  average  fractional  co- 
variance  (from  Equation  12)  is  plotted  (on  a  loga¬ 
rithmic  scale)  against  the  percentage  of  full  band¬ 
width  that  the  nodal  communications  resource  can 
support.  Coincidental  plots  for  one  of  the  fighters,  one 
of  the  bombers,  and  the  advanced  fighter,  for  both 
the  communications  management  algorithms,  round 
robin  (RR)  and  information  based  (IN),  are  provided 
on  each  of  the  figures,  (a)  to  (c).  The  corresponding 
variance  plots  are  provided  in  Fig.  5  with  y  axes  rang¬ 
ing  from  0.001  to  10.  We  refer  to  these  plots  as  process 
models  which  relate  the  performance  of  the  communi¬ 
cations  management  algorithms  to  the  percentage  of 
full  bandwidth  employed  for  inter-platform  communi¬ 
cations. 

7.  Discussion 

This  section  discusses  the  results  in  more  detail.  In 
addition,  a  qualitative  comparison  of  the  results  are 
described  with  a  specific  application  to  battlespace 
scenarios. 

Two  process  model  metrics  describe  the  perfor¬ 
mance  of  the  communications  management  algo¬ 
rithms.  They  are  the  average  fractional  covariance 
volume,  IP  xx  [/|P req  |,  and  its  associated  standard  de¬ 
viation.  The  latter  is  comprised  of  a  very  small 
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(a)  Percentage  Error  in  Decison  Value  =  10 


(a)  Percentage  Error  in  Decison  Value  =  10 


(b)  Percentage  Error  in  Decison  Value  =  50 


(c)  Percentage  Error  in  Decison  Value  =  90 


Figure  5:  Standard  Deviation  Results  (10  Monte 
Figure  4:  Average  Results  (10  Monte  Carlo  ‘runs’)  Carlo  ‘runs’) 
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random  component  (introduced  by  having  the  sensor 
probability  of  detection,  Pd,  in  the  simulations  set  to  a 
value  less  than  unity)  and  additionally,  for  the  round- 
robin  algorithm,  by  a  random  ordering  of  the  targets. 
It  is  also  comprised  of  a  systematic  component.  This 
is  a  function  of  the  time-division-multiple-access  com¬ 
munications  protocol  in  which  each  node  follows  a 
staggered  sequence  of  . .  .transmit-receive-receive. . .. 

Note  that  for  the  round  robin  and  information  based 
communications  management  algorithms,  the  average 
and  standard  deviation  metrics  exhibit  a  greater  dif¬ 
ference  margin  for  lower  percent  error  in  decision  val¬ 
ues.  In  fact  we  predict  that  as  the  percent  error  in 
decision  value  approaches  100%,  the  statistical  per¬ 
formance  of  each  algorithm  would  be  similar.  In  ad¬ 
dition,  on  an  individual  target  basis,  the  information 
based  algorithm  performs  better  than  the  round  robin 
algorithm  on  the  advanced  fighter  target,  while  the 
situation  is  reversed  for  the  bomber  aircraft.  A  com¬ 
parison  of  the  curves  for  the  fighter  aircraft  shows  that 
the  relative  performance  of  the  communications  man¬ 
agement  algorithms  is  sensitive  to  the  percentage  of 
the  full  bandwidth  that  is  being  employed.  This  is 
a  consequence  of  the  information  based  communica¬ 
tions  management  algorithm  allocating  more  overall 
bandwidth  to  the  targets  that  are  difficult  to  track. 

We  now  consider  the  application  of  these  process  mod¬ 
els  to  a  system  design  example.  Let  us  assume  that  the 
scenario  described  in  Section  5.2  has  a  specification 
requirement  that  all  targets  should  have  an  average 
covariance  volume  of  less  than  |Pre?|  with  an  inter¬ 
platform  bandwidth  of  60%  that  required  to  commu¬ 
nicate  all  the  available  data.  For  this  example,  the 
system  design  decision  is  to  determine  what  commu¬ 
nications  management  algorithm  to  employ. 

A  sample  of  possible  solutions  have  been  generated 
from  the  results  presented  in  Section  6  and  illustrated 
in  Fig.  4.  Here  the  average  performance  metric  (shown 
in  Fig.  6)  associated  with  the  advanced  fighter  is 
considered  since  its  value  is  larger  than  that  for  the 
bomber  and  fighter  aircraft,  in  a  range  around  60%  of 
full  bandwidth.  From  the  results  generated  in  our  in¬ 
vestigation  two  possible  communications  management 
options  are  available:  using  the  round  robin  algorithm, 
or  using  the  information  based  algorithm.  The  latter 
algorithm  was  subject  to  10,  50  or  90  percent  errors 
in  the  decision  values.  Since,  the  system  specification 
has  to  meet  the  required  covariance,  Pre?,  a  value  of 
|P |/|P req |  >  1  indicates  when  the  specification  is 
not  met  and  a  value  of  |P**!/|Pre?|  <  1  indicates 
when  the  specification  has  been  met.  Here,  the  round 
robin  algorithm  does  not  meet  the  specification  pro¬ 
ducing  a  volume  of  uncertainty  that  is  approximately 
twice  the  requirement,  see  Fig.  6.  Likewise  the  in¬ 
formation  based  algorithm  with  a  90  percent  error 
in  decision  value  also  fails,  although  not  so  badly  as 
the  round  robin  algorithm.  The  information  based 


algorithm  with  a  50  percent  error  in  decision  value 
achieves  the  specification.  This  would  probably  be 
chosen  in  preference  to  the  10  percent  error  in  decision 
value  implementation  as  it  would  be  less  computation¬ 
ally  expensive,  requiring  a  less  rigorous  algorithm  for 
predicting  the  receiver’s  information  (see  equations  9 
and  10).  This  would  release  valuable  computation  for 
other  processes1 . 

This  is  an  example  of  applying  the  communications 
management  process  models  developed  during  our 
investigation  to  meeting  specification  requirements 
while  preserving  a  reasonable  computational  require¬ 
ment. 


Figure  6:  Application  Example  (Communications 
bandwidth  =  60  %  of  Full  Bandwidth,  Performance 
requirement  =  Pre9). 

8.  Conclusions  and  Summary 

Algorithms  for  managing  a  constrained  communica¬ 
tions  resource  in  a  decentralised  target  tracking  sys¬ 
tem  have  been  investigated.  In  a  simulated  multi¬ 
target  and  multi-platform  battlespace  scenario,  the 
inter-platform  communications  bandwidth  was  varied 
between  two  extremes.  At  one  end  of  the  spectrum 
the  bandwidth  was  able  to  support  state  information 
transmission  on  all  the  targets  at  every  communica¬ 
tion  cycle.  At  the  other  end  of  the  spectrum  it  was 
unable  to  support  the  transmission  of  any  information 
about  the  states  of  the  targets.  Results  were  also  ob¬ 
tained  for  a  range  of  intermediate  bandwidth  values. 
We  presented  our  results  in  a  form  that  is  useful  in 
terms  of  gauging  whether  the  management  algorithms 
allow  the  system  to  meet  some  arbitrary  system  re¬ 
quirement.  The  results  indicate  that  a  system  which 
implements  the  information  theoretic  communications 
management  algorithm  is  better  (in  terms  of  produc¬ 
ing  smaller  average  state  error  ellipsoids  around  its 
tracks)  than  a  system  which  employs  the  round-robin 
algorithm.  The  degree  of  improvement  depends  on  the 

1  Alternatively,  the  more  cautious  designer  may  decide  to  im¬ 
plement  the  10  percent  error  in  decision  value  algorithm  allow¬ 
ing  a  larger  margin  of  error  between  the  specification  require¬ 
ment  and  that  given  by  the  process  model. 
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percentage  of  the  full  bandwidth  that  is  being  used, 
and  is  more  evident  for  the  highly  manoeuvring  tar¬ 
gets.  The  decision  on  whether  or  not  to  employ  the  in¬ 
formation  theoretic  communications  management  al¬ 
gorithm  will  depend  on  a  number  of  factors.  The  crit¬ 
ical  factors  are  likely  to  be  the  amount  of  bandwidth 
that  is  available,  the  computing  resources  on  the  plat¬ 
forms,  the  nature  of  the  battlespace  scenario  (particu¬ 
larly  the  number  and  the  mix  of  target  types),  and  the 
predicitve  power  of  platforms  in  the  network  relative 
to  other  platforms. 
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1.  SUMMARY 

Through  the  NATO  Alliance  Ground  Surveillance  (AGS) 
Project,  NATO  is  proposing  to  procure  an  airborne  ground 
surveillance  system  based  on  a  NATO-owned  and  operated 
core  capability  which  would  be  supplemented  by  inter¬ 
operable  national  assets,  as  required.  The  NATO 
Consultation,  Command  and  Control  (C3)  Agency  (NC3A), 
together  with  support  from  six  nations,  France,  Germany, 
Italy,  Norway,  The  United  Kingdom  and  The  United  States, 
has  established  a  testbed  capability  which  includes 
simulations  of  the  candidate  sensor  systems  and  various 
exploitation  capabilities.  This  testbed,  coupled  with  various 
other  command  and  control  testbeds  resident  at  NC3A,  allows 
the  study  of  many  aspects  of  command  and  control  associated 
with  the  AGS  problem. 

Essential  to  the  selected  ownership  option  is  the  concept  of 
the  interoperability  of  the  NATO-owned  assets  with  each 
other  and  with  the  nationally-owned  assets.  In  order  to 
address  this  problem,  the  NC3A,  in  conjunction  with  the  six 
nations  has  developed  an  architecture  and  a  standard  format 
for  the  sensor  data,  so  that  any  ground  station  could  be  made 
to  receive  and  process  data  from  any  of  the  candidate 
platforms.  Important  to  the  solution  is  the  ability  to 
simultaneously  receive  and  process  data  from  different  sensors 
and  sensor  platforms  allowing  for  the  fusion  of  data  and 
potentially  leading  to  improved  surveillance  coverage, 
tracking  performance  and  accuracy.  This  approach  has  been 
prototyped  and  tested  in  the  laboratory  utilizing  the  AGS 
testbed.  Recently,  a  large  experiment  was  conducted  during 
which  several  real  AGS  platforms  and  ground  stations  were 
utilized  in  order  to  prove  the  interoperability  concepts  that 
have  been  developed  at  NC3A  in  a  more  practical 
environment. 

In  order  to  set  the  stage,  the  paper  begins  with  a  brief 
overview  of  the  NATO  AGS  project  followed  by  a  definition 
of  the  interoperability  problem.  Next,  the  hardware  and  data 
architecture  developed  at  NC3A  for  achieving  interoperability 
is  described.  Finally,  the  paper  concludes  with  a  description 
of  a  recent  AGS  interoperability  experiment  used  to  prove  the 
concepts  is  presented. 

2.  NATO  AGS  PROJECT  BACKGROUND 

The  idea  of  using  airborne  platforms  to  provide  surveillance 
information  of  enemy  troop  positions  and  movements  is  not  a 
new  one.  Since  the  dawn  of  the  age  of  manned  flight  Armies 
have  been  exploiting  the  use  of  airborne  observation  to  look 
deep  beyond  the  front  lines.  In  as  early  as  the  summer  of 
1911  the  concept  was  being  applied  [Ref.  1]  and  by  the  end  of 


the  First  World  War  it  had  been  refined  and  well  developed. 
However,  until  after  the  end  of  Second  World  War,  the 
observation  techniques  were  largely  based  on  optical  means 
such  as  observers  or  photographic  cameras  and  were  thus 
limited  in  their  ability  to  “see”  to  times  of  fair  weather  and 
daylight.  The  sensors  provided  only  limited  range  which 
meant  that  only  limited  observations  were  possible  and  that 
the  platforms  had  to  penetrate  enemy  lines  in  order  to  make 
their  observations  exposing  them  to  enemy  fire.  It  was  only 
during  the  cold  war  that  technology  had  advanced  to  the  point 
where  one  could  start  to  think  of  using  alternate  sensors. 
Recent  advances  in  technology  have  made  it  possible  to  push 
the  airborne  ground  surveillance  concept  in  several  important 
directions.  Advances  in  sensor  and  computing  technology 
allow  observation  to  much  greater  ranges  during  all  types  of 
weather  at  any  time  of  the  day  or  night.  As  well,  because  of 
their  increased  range,  the  coverage  can  be  over  a  much  greater 
area  providing  a  complete,  theatre-wide,  situational  awareness 
as  well  as  target  selection  capability.  Because  of  the  advances 
in  computer  processing  power  and  communication  systems 
the  information  gathered  can  be  delivered  in  an  immediate 
manner,  giving  commanders  access  to  a  continuous  and 
current  picture  of  the  battlefield. 

In  April  1993  NATO  formed  an  Ad  Hoc  Multi-Service  Group 
(MSG)  to  investigate  the  requirements  for  a  NATO  Alliance 
Ground  Surveillance  (AGS)  capability  [Ref.  2],  Following 
this  work,  an  Embryonic  Project  Structure  (EPS)  was 
established  consisting  of  an  Embryonic  Project  Office  (EPO) 
and  a  Steering  Committee.  The  EPS’  work  was  divided  into 
two  phases.  During  the  first  phase  the  ownership  options  for  a 
NATO  AGS  capability  were  to  be  investigated  and  during  the 
second  phase  a  NATO  Staff  Requirement  (NSR)  and  other 
documents  which  will  define  the  NATO  AGS  programme 
were  to  be  drafted. 

The  first  phase  of  the  EPS’  work  concluded  in  the  fall  of  1995 
and  led  to  the  endorsement  by  the  North  Atlantic  Council 
(NAC),  at  the  Ministerial  level,  of  the  concept  of  a  “NATO- 
owned  and  operated  core  capability  supplemented  by 
interoperable  national  assets.”  The  exact  composition  of  this 
NATO  owned  core  is  not  known  at  this  time  but  could  consist 
of  both  fixed  wing  and  rotary  wing  surveillance  platforms. 
The  EPS  has  been  reconstituted  since  November  1 996  as  the 
Provisional  Project  Structure  and  is  at  present  in  the  midst  of 
their  second  phase  of  work.  A  decision  is  expected  in  the 
autumn  as  to  whether  to  proceed  with  immediate  procurement 
due  to  the  urgency  of  the  need  or  to  take  a  more  deliberate 
approach  and  study  the  problem  more  fully. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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Four  systems  are  being  studied  as  primary  candidates  for  the 
core  capability,  namely:  the  Italian  Complesso  Radar 
Eliportato  Per  La  Sorveglianza  (CRESO),  the  French 
Helicoptere  d’Observation  Radar  et  d’ Investigation  sur  Zone 
(HORIZON),  the  United  States’  Joint  Surveillance  and  Target 
Attack  Radar  System  (JSTARS)  and  the  United  Kingdom’s 
Airborne  Stand-Off  Radar  (ASTOR)  systems.  All  of  these 
systems  consist  of  an  airborne  sensor  platform  and  ground- 
based  exploitation  stations.  The  airborne  platforms 
communicate  with  the  ground  stations  over  system-specific 
medium/high-bandwidth  data  links.  Complementary  assets 
could  range  from  current  reconnaissance  assets,  UAVs, 
satellites,  etc. 

The  fixed  wing  platforms  both  offer  a  Synthetic  Aperture 
Radar  (SAR)  sensor  which  is  capable  of  detailed  imaging  of 
the  ground  at  considerable  stand-off  distances  and  the 
detection  of  fixed  targets.  These  SAR  sensors  will  have  to 
operate  in  either  a  high  resolution  Spot  mode,  where  a  small 
area  is  imaged  in  detail,  and  a  wider-area,  lower  detail,  Swath 
mode.  As  well,  these  platforms  offer  a  Moving  Target 
Indicator  (MTI)  radar  mode.  This  mode  facilitates  the 
detection  of  any  targets  which  are  in  motion  at  or  near  the 
surface  of  the  earth,  again,  at  a  considerable  stand-off 
distance.  The  rotary-wing  platforms  each  have  MTI  sensors, 
but  no  SAR  capability.  The  complementary  assets  could 
provide  a  variety  of  other  sensor  data,  including  ESM, 
Electro-Optic,  Infra-Red,  etc. 

3.  THE  INTEROPERABILITY  PROBLEM 

Although  the  interoperability  problem  could  be  considered  in 
its  entirety,  it  has  generally  been  divided  into  two 
components:  Intra-AGS  Interoperability  and  Extra-AGS 
Interoperability.  By  Intra-AGS  Interoperability  is  meant  the 
interoperability  between  all  of  the  various  NATO  owned  and 
non-NATO  owned  AGS  and  other  complementary 
surveillance  assets.  That  is,  Intra-AGS  Interoperability  deals 
with  the  problem  of  how  to  get  all  available  sensor  data  to  the 
ground  stations  in  such  a  way  that  it  can  be  exploited  and 
possibly  fused  with  data  from  other  sensors  and  intelligence 
sources.  By  Extra-AGS  Interoperability  is  meant  the 
integration  of  the  AGS  capability  and  AGS  data  with  the 
existing  and  planned  NATO  and  national  C3I  infrastructures. 
The  Extra-AGS  interoperability  provides  the  channels  and 
means  to  share  the  AGS  data  with  the  end  users  and  to  import 
data  into  AGS  that  can  assist  in  the  exploitation  process. 
These  two  aspects  of  interoperability  are  illustrated  in  Figure 
1 .  This  report  deals  primarily  with  the  first  issue,  that  is,  the 
interoperability  between  various  AGS  assets  and  does  not 
address  integration  of  the  AGS  data  into  the  larger  NATO  C3I 
infrastructure  in  great  detail. 

There  have  been  many  studies  in  the  past  which  have 
addressed  the  issue  of  interoperability  of  the  Alliance  Ground 
Surveillance  capability  specifically  or  airborne  reconnaissance 
systems  in  general.  Some  of  these  are  listed  in  the  references 
at  the  end  of  this  report  [Refs.  3-10],  NATO  Air  Group  IV, 
through  efforts  such  as  STANAGs  7023  and  7085  have  been 
working  to  achieve  standards  for  data  formats  and  data  links 
in  order  to  facilitate  information  exchange.  The  approach 
taken  in  this  report  is  to  construct  an  architecture  which 
allows  enough  flexibility  that  it  can  capitalize  on  these 
standardisation  efforts  as  they  become  available,  but  is  not 
dependant  on  them  for  its  success.  Because  of  the  existence 
of  legacy  national  systems,  the  mandating  of  any  one  standard 
or  STANAG  with  which  the  NATO  owned  capability  must  be 


compliant  is  not  considered  to  be  a  complete  solution  to  the 
problem. 

4.  PROPOSED  ARCHITECTURE  FOR  INTRA-AGS 
INTEROPERABILITY 

This  section  describes  a  proposed  architecture  for  the  solution 
of  the  Intra-AGS  Interoperability  problem.  In  the  first  sub¬ 
section,  the  proposed  hardware  architecture  is  described.  Next 
the  data  that  is  to  be  interchanged  is  identified  and  finally  the 
format  for  the  data  is  discussed.  These  three  components 
together  constitute  a  complete  architectural  solution  to  the 
problem. 

4.1  The  Proposed  Hardware  Architecture 

A  generic  AGS  capability,  as  presently  configured,  is 
illustrated  in  Figure  2.  It  consists  of  an  airborne  sensor 
platform,  a  native  data  link  specific  to  the  particular  platform, 
a  Ground  Data  Terminal  (GDT)  for  receiving  the  data  on  the 
ground  and  an  Ground  Exploitation  Capability  (GEC)  for 
converting  the  data  into  information  useful  to  the  operational 
commander. 

Although  the  configuration  of  the  GEC  for  each  candidate 
system  is  different,  they  all  perform  a  number  of  generic 
functions,  such  as  display  and  exploitation  of  MTI  data, 
display  of  cartographic  information,  visualization  of  SAR 
information,  etc.  The  GDT  and  GEC  together  form  what  is 
commonly  called  a  Ground  Station  (GS)  or  Ground  Station 
Module  (GSM).  Thus  the  ground  station  consists  of  two 
components:  a  system  specific  GDT,  unique  to  each  of  the 
candidate  systems;  and  a  GEC,  which  is  certainly  different  for 
each  system  but  which  performs  many  of  the  common, 
generic  functions  required  for  exploitation  of  AGS  data. 

Analysis  has  shown  that  the  optimum  solution  [Ref.  1 5]  to  the 
interoperability  problem,  from  a  purely  operational 
perspective,  is  to  have  all  platforms  utilize  a  common  data 
link  and  common  data  formats.  This  allows  the  operational 
commander  maximum  flexibility  in  the  employment  of  the 
surveillance  assets.  Unfortunately,  when  one  includes 
technical  and  cost  considerations  and  consideration  of  legacy 
systems,  this  solution  does  not  appear  to  be  practical  in  the 
near-term. 

An  alternate  solution,  which  shares  many  of  the  operational 
and  other  benefits  of  the  common  data  link  solution,  proposes 
that  the  two  components  to  the  GS  be  de-coupled  so  that  one 
can  think  of  the  GDT  and  GEC  as  independent  entities, 
connected  together  through  a  standardized  common  data 
format.  This  is  the  key  concept  of  the  proposed  architecture 
and  is  illustrated  in  Figure  3.  By  adopting  such  an  approach, 
the  possibilities  illustrated  in  Figure  4  become  possible.  One 
can  think  of  combining  any  GDT  with  any  available  GEC.  It 
is  thus  possible  to  deploy  one  GEC  and  several  GDTs  in  order 
to  receive  data  from  any  AGS  or  other  surveillance  asset 
operating  in  an  area.  The  architecture  also  makes  it  possible 
to  attach  several  GDTs  to  the  same  GEC  and  thus  achieve 
reception  of  data  from  several  airborne  assets  simultaneously. 
This  capability  allows  for  the  possibility  of  real-time  fusion  of 
data  from  multiple  sensors  which  could  lead  to  advantages 
such  as  improved  detection,  increased  surveillance  coverage, 
improved  track-continuity,  etc.  [Ref.  16].  The  architecture 
can  accommodate  SATCOM  data  links  or  standardized 
common  data  links  when  they  become  readily  available  and 
commonly  used.  Finally,  the  proposed  architecture  provides  a 
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clear  and  relatively  inexpensive  upgrade  path  for  legacy 
systems.  By  providing  an  input  for  data  in  this  common 
format,  the  legacy  ground  stations  can  be  made  to  receive  data 
from  other  sensors.  Any  NATO-owned  ground  stations  could 
receive  data  from  national  assets  utilizing  the  appropriate 
GDT  and  the  common  interface  format.  A  very  important 
point  is  that  the  architecture  requires  no  expensive 
modifications  to  existing  airframes  -  the  changes  are  all 
implemented  in  the  ground  segment. 

4.2  The  Proposed  Data  Types 

The  hardware  architecture  for  Intra-AGS  Interoperability  has 
been  established  in  the  previous  section.  In  this  section  the 
data  types  that  should  be  used  within  the  architecture  are 
established.  This  is  done  by  first  looking  at  data  in  a  generic 
sensor  system  and  then  looking  at  the  data  existing  in  the 
airborne  and  ground  segments  of  the  candidate  AGS 
platforms. 

4.2. 1  Generic  Surveillance  Sensor  Data  Flow 
Data  can  exist  in  a  number  of  different  forms  within  AGS  or 
other  surveillance  systems,  ranging  from  raw  sensor  data 
through  processed  sensor  data  and  finally  to  fully  exploited 
data  which  results  in  useful  operational  information. 

By  raw  sensor  data  is  meant  the  data  available  directly  at  the 
output  of  the  sensor  sub-system.  For  radar  systems  this  is 
usually  in  the  form  of  a  digitised  video  signal  (time  series).  In 
the  case  of  SAR  radar  data,  this  would  equate  to  data  prior  to 
processing  in  the  SAR  processor  where  the  SAR  image  is 
formed  (so  called  I  and  Q  channels).  In  the  case  of  ESM,  this 
again  would  be  a  sampled  time  series.  Depending  on  the 
particular  sensor,  area  of  coverage,  resolution,  etc.,  the  precise 
bandwidth  requirements  for  raw  sensor  data  can  vary  but  are 
in  the  order  of  hundreds  of  megabits  per  second. 

By  processed  sensor  data  is  meant  sensor  data  that  has  been 
manipulated  in  a  signal  processor  in  order  to  make  it  of  use  to 
an  analyst.  In  the  case  of  MTI  data  this  would  be  the  radar 
data,  after  it  has  undergone  detection  processing.  In  its 
simplest  form,  this  process  is  accomplished  by  comparing  the 
data  to  a  threshold,  and  declaring  a  target  present  if  the  signal 
exceeds  the  threshold.  Further  information  is  then  extracted 
from  the  data,  such  as,  position,  radial  velocity,  and  perhaps 
some  idea  of  target  classification.  In  the  case  of  SAR  data,  the 
intermediate  step  is  the  formed  SAR  image,  usually  consisting 
of  an  array  of  geo-located  pixel  values.  At  this  stage  the 
images  would  also  be  corrected  for  any  airborne  platform 
motions  and  focused  in  order  to  increase  the  sharpness  of  the 
images.  Precise  geo-location  information  is  added  to  the 
image  so  that  in  subsequent  stages  accurate  target  location  and 
mensuration  information  can  be  extracted.  Data  at  this 
intermediate  stage  is  referred  to  as  pre-exploitation  data.  For 
pre-exploitation  data,  the  bandwidth  required  is  on  the  order 
of  tens  of  megabits  per  second. 

Finally,  after  the  exploitation  stage,  the  data  is  transformed 
into  information  on  which  commanders  can  base  their 
decisions.  For  MTI  this  could  take  the  form  of  intelligence 
reports  or  target  tracks,  giving  position,  true  velocity, 
identification,  etc.  This  is  achieved  by  correlating  consecutive 
“looks”  with  the  MTI  sensor  to  form  tracks.  MTI  detections 
can  also  be  grouped  together  to  be  identified  as  convoys. 
Targeting  information  such  as  predictions  of  future  positions 
can  be  made  and  MTI  data  can  be  combined  with  other 


sources  of  information  such  as  order  of  battle  data  bases, 
logistics  movement  plans,  etc.  in  order  to  identify  each  target 
of  interest,  etc.  For  SAR  data,  the  exploited  version  could  be 
an  annotated  image  or  simply  a  list  of  fixed  targets,  including 
their  location,  identification,  etc.  These  operations  at  present, 
unlike  the  operations  in  the  first  two  stages  of  processing,  rely 
heavily  on  a  skilled  and  experienced  operator  and  are 
accomplished  at  the  exploitation  stations,  either  aboard  the 
airborne  platform  or  within  the  ground-based  exploitation 
station.  It  is  the  operationally  significant  information  which 
results  from  the  exploitation  that  we  shall  refer  to  as  post¬ 
exploitation  data.  The  data  rates  required  can  vary  greatly 
depending  on  the  exact  nature  of  the  information  being 
transmitted  and  the  timeliness  requirements.  In  general  this 
can  be  on  the  order  of  tens  or  hundreds  of  kilobits  per  second 

These  stages  are  illustrated  in  Figure  5. 

4.2.2  Sharing  Pre-  vs  Post  Exploitation  Data 

At  the  highest  level,  interoperability  can  be  achieved  in  one  of 
three  ways,  namely:  raw  sensor  data  can  be  shared;  data  can 
be  shared  pre-exploitation;  or  data  can  be  shared  post¬ 
exploitation.  First  comparing  the  last  two  possibilities,  the 
sharing  of  unexploited  data  results  in  a  higher  accuracy  and 
more  continuous  tracking  of  targets  [Ref.  1 6]  than  the  sharing 
of  exploited  data.  The  data  from  several  sensors  may  be 
complementary  and  gaps  occuring  in  any  single  sensor 
coverage  (due  to  terrain  screening  or  doppler  blind  problems) 
may  be  filled  in  by  the  fusion  of  data  from  multiple  sources  so 
that  a  more  continuous  track  is  achievable.  If  tracks  are 
established  on  the  basis  of  data  from  individual  sensors  and 
then  fused  subsequently,  due  to  gaps  in  sensor  coverage  it 
may  not  be  possible  to  maintain  a  track  in  certain 
circumstances.  However,  sharing  unexploited  data  requires 
higher  bandwidth  communication  channels  for  transferring  the 
data  from  all  sensors  to  the  exploitation  stations,  rather  than 
just  passing  low  bandwidth  exploited  data  between 
exploitation  capabilities. 

4.2.3  Data  Types  Available  in  Candidate  AGS  Systems 

In  this  section,  the  availability  of  each  of  the  three  data  types 
is  discussed  for  the  four  AGS  candidate  systems. 

4.2.3. 1  ASTOR 

At  this  time  it  is  unclear  whether  the  ASTOR  system  will 
provide  raw  sensor  data  or  pre-exploitation  data  to  the  ground 
station.  In  either  case,  some  exploitation  will  be  possible  on 
board  the  aircraft.  For  the  case  of  ASTOR  then,  in  the  worst 
case,  raw  sensor  data,  pre-exploitation  data  and  post¬ 
exploitation  data  exist  in  the  airborne  platform  and  on  the 
ground. 

4.23.2  CRESO 

The  CRESO  system  transmits  only  pre-exploitation  data  to  the 
ground.  Therefore,  for  CRESO,  raw  sensor  data  and  pre¬ 
exploitation  data  are  available  in  the  air  and  pre-exploitation 
and  post-exploitation  data  are  available  on  the  ground. 

4.2.33  HORIZON 

HORIZON  has  essentially  the  same  exploitation  capability  in 
the  airborne  platform  as  is  available  in  the  ground  station. 
Therefore,  for  HORIZON,  raw  sensor  data,  pre-exploitation 
and  post-exploitation  data  are  available  in  the  air.  On  the 
ground,  as  HORIZON  links  only  pre-exploitation  data  to  the 
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ground  station,  pre-exploitation  and  post  exploitation  data  are 
available. 

4. 2. 3. 4  Joint  STARS 

Joint  STARS  has  an  extensive  airborne  exploitation  capability 
so  that  raw  sensor  data,  pre-exploitation  data  and  post 
exploitation  data  are  all  available  in  the  aircraft.  On  the 
ground,  as  the  data  link  carries  only  pre-exploitation  data,  pre¬ 
exploitation  and  post-exploitation  data  are  available. 

4. 2. 3. 5  Other,  Complementary  Sensors 

A  variety  of  other  sensor  types  might  be  brought  to  bear  on 
the  ground  surveillance  problem.  .  These  include  not  only  the 
ones  discussed  in  some  detail  such  as  SAR  and  MTI,  but  also 
could  include  ESM,  ECCM,  Electro-Optic  (EO),  Long-Range 
Oblique  Photography  (LOROP),  Infra-Red  (IR),  etc. 
Depending  on  the  sensor  and  implementation,  data  sent  to  the 
ground  can  be  either  raw  sensor  data  or  pre-exploitation  data. 

4.2.4  Summary 

For  the  proposed  hardware  architecture,  it  can  be  concluded 
that  there  is  no  point,  at  this  time,  in  considering  the  sharing 
of  raw  sensor  data  as  three  of  the  four  candidate  AGS  systems 
do  not  send  this  data  to  the  ground.  The  sharing  of  post¬ 
exploitation  data  could  be  considered;  however,  this 
presupposes  that  the  data  from  each  sensor  be  individually 
exploited.  This  approach  offers  no  savings  in  terms  of  sharing 
exploitation  stations.  As  well,  this  solution  suffers  from  the 
problems  discussed  above  related  to  track  accuracy  and 
continuity.  It  is  therefore  concluded  that  the  preferred  data  to 
be  shared  is  pre-exploitation  data. 

4.3  Data  Formats 

At  present,  there  exist  a  number  of  STANAGS  and  other 
standards  which  define  data  formats  that  could  be  used  to 
represent  the  data  within  an  AGS  system  [Refs.  6,  12,  13,  14, 
15].  However,  there  exists  no  standard  which  can 
satisfactorily  represent  pre-exploitation  data.  Raw  sensor  data 
can  be  adequately  represented  in  the  format  specified  in 
STANAG  7023  [Ref.  6].  Post-exploitation  data  can  be 
represented  in  the  formats  specified  in  STANAG  3377  [Ref. 
12],  STANAG  5516  [Ref.  13],  ADatP-3,  [Ref.  14],  STANAG 
4545  [Ref.  15]  and  others.  Therefore  a  new  set  of 
standardized  formats  are  required  to  represent  pre-exploitation 
data.  The  work  at  NC3A  over  the  last  several  years  has 
focused  on  developing  and  implementing  formats  suitable  for 
this  purpose  [Ref.  18],  Discussions  with  Air  Group  IV  have 
commenced  with  a  view  to  incorporating  these  formats  into  a 
future  release  of  the  STANAG  7023. 

4.4  Proposed  Architecture  Summary 

In  summary,  the  proposed  architecture  consists  of  three  parts: 
the  hardware  architecture  based  on  the  concept  of  de-coupling 
of  the  GDT  and  GEC  and  insertion  of  an  intermediate 
standard  common  data  format;  the  sharing  of  pre-exploitation 
data;  and  the  use  of  the  formats  developed  at  NC3A  for  data 
representation.  These  concepts  have  been  developed, 
prototyped  and  tested  in  the  NC3A  AGS  Testbed  over  the  past 
several  years.  The  final  proof  of  concept  occurred  in  June 
1997  when  an  interoperability  experiment  was  held  outside  of 
Paris.  This  experiment  is  described  in  the  following  section. 


5.  PARIS  INTEROPERABILITY  EXPERIMENT 

The  Paris  Interoperability  Experiment  (PIE)  was  a  large 
experiment  involving  over  200  people  from  six  nations  and 
NC3A.  It  was  held  at  the  Centre  d-Essais  en  Vol  (CEV) 
Bretigny-sur-Orge,  South  of  Paris,  during  the  week  following 
the  1997  Paris  Air  Show.  The  experiment  is  described  in 
terms  of  its  aims,  conduct  and  achievements  in  the  following 
paragraphs. 

5.1  PIE  Aims 

There  were  two  major  aims  of  PIE.  The  first  was  to  prove  that 
the  interoperability  concepts  described  above  could  be  made 
to  work  in  a  practical  environment.  They  had  been  prototyped 
and  tested  in  a  laboratory  environment  and  had  reached  a 
sufficient  state  of  maturity  that  it  was  time  to  take  them  out  in 
the  field  and  prove  them  in  practice.  The  second  major  aim 
was  to  gather  a  data  set  that  could  be  used  for  analysis  of  the 
complementary  nature  of  AGS  assets.  For  the  first  time  there 
would  be  the  possibility  of  gathering  data  including 
simultaneous  coverage  of  the  same  terrain  by  two  or  more 
AGS  platforms.  Specifically  the  aims  were: 

5.1.1  Prove  Interoperability  Concept 

The  specific  test  conducted  were  used  to  test  the  following: 

a.  the  transfer  of  MTI  in  real-time  from  Joint  STARS  to  the 
HORIZON  and  other  ground  station; 

b.  the  transfer  of  MTI  in  real-time  from  HORIZON  to  the 
Joint  STARS  and  other  ground  stations; 

c.  the  transfer  of  SAR  data  from  Joint  STARS  to  HORIZON 
and  other  ground  stations; 

d.  the  simultaneous  real-time  transfer  of  SAR  and  MTI  data 
from  Joint  STARS  and  MTI  data  from  HORIZON  to  all 
ground  stations; 

e.  the  transfer  of  Radar  Service  Requests  (RSR)  from  each 
ground  station  to  each  of  the  airborne  platforms; 

f.  understand  the  Electro-Magnetic  Compatibility  (EMC) 
problems  related  to  the  simultaneous  operation  of  multiple 
data  links  in  close  proximity;  and 

g.  transfer  of  secondary  (post-exploitation)  data  between 
ground  stations  and  between  ground  stations  and  end- 
users.  This  included  the  transfer  of  Link- 16  formatted 
messages,  ADatP-3  messages  and  messages  describing 
locations  and  other  characteristics  of  detected  convoys. 

5.1.2  Gather  Multi-Platform  Data  Set 

The  aircraft  orbits  and  convoy  routes  were  established  in  order 
to  gather  a  data  set  which  could  be  used  to  analyze  and 
quantify  the  following  measures  of  performance  related  to  the 
complementary  use  of  multiple  AGS  platforms: 

a.  the  achievable  improvements  in  the  detection  of  moving 
targets; 

b.  the  achievable  improvements  in  target  resolution; 

c.  the  achievable  improvements  in  target  classification; 

d.  the  achievable  improvements  in  track  continuity, 
compensating  for  gaps  in  coverage  due  to: 

i.  zero  target  Doppler  situations; 

ii.  terrain  masking;  and 
ii.  coverage  area. 

5.2  PIE  Conduct 

The  PIE  Experiment  was  conducted  from  23  to  27  June  at 
CEV.  Prior  to  these  dates  all  equipment  was  brought  to 
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Bretigny.  During  the  first  day  of  flying,  each  platform  was 
used  individually  for  three  hours  in  order  to  conduct  tests 
related  to  the  ability  of  single  platforms  to  distribute  the  data 
in  accordance  with  the  prototype  architecture.  The  remainder 
of  the  time  involved  simultaneous  flights  of  the  two  platform 
types. 

Three  target  convoys  were  employed  for  the  first  three  days  of 
PIE.  The  convoys  were  furnished  by  the  French  Army  and 
consisted  of  seven  vehicles  each.  The  lead  vehicle  was  soft- 
skinned  while  the  remaining  six  were  wheeled  armoured 
vehicles.  During  the  last  day  of  flying,  a  single  convoy  of 
similar  composition  was  used.  The  convoy  routes  involved  a 
variety  of  terrain  types,  including:  relatively  flat  terrain 
presenting  few  visibility  problems;  more  challenging  terrain 
presenting  some  screening  problems  from  certain  directions; 
and  quite  challenging  terrain. 

The  convoys  were  equipped  with  Global  Positioning  System 
(GPS)  receivers.  The  positions  of  the  convoys  were  relayed 
via  a  commercial  satellite  system  known  as  EuTelTracs,  back 
to  CEV.  This  provided  real-time  updates  of  the  convoy’s 
positions  for  monitoring  and  control  purposes  during  the 
experiment  and  also  truth  data  for  post  experiment  analysis. 

The  equipment  was  set-up  as  indicated  in  Figure  6. 
Participating  in  the  experiment  were  one  United  States  Joint 
STARS  E-8  aircraft  and  two  Joint  STARS  GSMs,  two  French 
HORIZON  helicopters  and  two  HORIZON  GSMs,  software 
and  hardware  representative  of  the  ground  stations  of  the 
United  Kingdom  ASTOR,  Italian  CRESO  and  German 
Domier  system.  Additionally,  the  Norwegian  Mobile  Tactical 
Operations  Centre  (MTOC),  the  NC3A  Prototype  Ace 
Intelligence  System  (PAIS)  and  Recognized  Air  and  Surface 
Picture  (RASP)  display  system  were  included  as 
representative  of  users  in  the  Army,  Intelligence  and  Air  Force 
communities. 

5.3  PIE  Achievements 

In  short,  all  of  the  technical  aims  of  PIE  were  met.  The  data 
from  both  HORIZON  and  Joint  STARS  was  exchanged  and 
also  distributed  to  the  other  participants  in  real-time  and 
simultaneously.  RSRs  were  sent  to  both  platforms  and 
serviced.  There  were  no  ECM  problems  experienced  during 
the  experiment.  All  three  data  links  that  were  operating  (two 
Joint  STARS  Surveillance  and  Control  Data  Link  (SCDL)  and 
one  HORIZON  Agatha  data  link)  in  the  area  functioned 
correctly  and  simultaneously. 

In  terms  of  the  second  aim,  that  is  the  gathering  of  a  data  set, 
16  hours  of  Joint  STARS  MTI  and  SAR  data  and  17  hours  of 
HORIZON  data  were  gathered.  Of  this  data  approximately  1 0 
hours  includes  overlapping  sensor  coverage,  in  the  following 
combinations: 

a.  8  hours  with  Joint  STARS  and  one  HORIZON; 

b.  1  hour  with  Joint  STARS  and  two  HORIZON;  and 

c.  1  hour  with  two  HORIZON. 

This  data  represents  a  very  unique  data  set  which  will  be  used 
as  a  basis  for  the  investigation  of  many  issues  in  the  future. 

6.  CONCLUSIONS 

This  report  has  presented  a  proposed  solution  to  the  problem 
of  Intra-AGS  interoperability  for  the  NATO  AGS  capability. 


The  approach  represents  a  complete  architecture  in  terms  of 
hardware,  data  types  and  data  formats.  This  architecture  has 
been  extensively  tested  in  the  laboratory  and  through  a 
significant  practical  experiment  in  the  field. 

It  is  considered  that  the  best  approach  to  the  achievement  of 
Intra-AGS  interoperability,  from  an  purely  operational 
perspective,  is  the  use  of  a  common  data  link.  However,  when 
one  also  takes  into  account  the  technical  and  related  cost  and 
risk  implications,  the  most  viable  approach,  in  the  short  to 
medium  term,  is  considered  to  be  to  share  the  pre-exploitation 
data  at  the  interface  between  the  GDT  and  the  GEC  through 
the  use  of  a  standardized,  common  format.  This  is  considered 
a  low  risk  solution  to  interoperability  which  will  allow 
tremendous  flexibility  in  the  employment  of  AGS  assets,  a 
clear  path  for  nations  to  follow  in  order  to  achieve 
interoperability  with  the  NATO  core  fleet  and  a  growth  path 
for  the  future. 
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Figure  1:  AGS  Interoperability 
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Figure  2:  Present  AGS  System  Configuration 


Figure  3:  Proposed  AGS  System  Configuration 
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SUMMARY 

Imagery  and  Signals  Intelligence  (1MINT 
and  SIGINT)  data  sources  and  processing  have 
traditionally  followed  independent  channels, 
with  cross  use  of  the  information  being  achieved 
only  by  manual  human  interaction  between  users 
at  the  end  of  their  respective  channels.  The 
characteristics  of  each  of  these  sources  are  quite 
different,  making  automatic  correlation  of  the 
data  a  challenging  task,  but  the  rewards  of 
correlated  data  hold  the  promise  for  more 
accurate  and  robust  surveillance  than  now 
available  from  either  independent  source. 
IMINT,  for  example,  is  characterized  by  a  low 
revisit  rate,  high  spatial  fidelity  and  broad  spatial 
context,  while  SIGINT  provides  a  high  revisit 
rate,  low  spatial  fidelity  and  a  broad  temporal 
context. 

This  paper  describes  the  specific  issues  that 
face  designers  who  seek  to  automatically  fuse 
imagery  and  signal  data  -  and  the  methods  used 
to  overcome  the  obstacles  posed  by  the 
fundamental  differences  in  the  data  types.  Four 
areas  of  IMINT-SIGINT  fusion  are  addressed:  1) 
fusion  for  cross-cueing  and  sensor  management, 
2)  fusion  for  improved  target  geolocation,  3) 
fusion  for  increased  target  detection  and 
identification  confidence,  and,  4)  fusion  for 
improved  visualization  of  the  battlefield.  The 
paper  provides  examples  illustrating  future 
solutions  using  terrain  data,  UAV  imaging 
sensors,  and  real-time-in-cockpit  (RT1C) 
techniques  illustrated  by  ERIM-developed 
technologies.  The  integration  of  these 
capabilities  into  joint  intelligence  systems  is 
illustrated  by  BTG-developed  technologies. 

The  paper  addresses  two  of  the  specific 
topics  listed  in  the  AGARD  call  for  papers:  1) 
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Applications  of  multiple  sensors  and  data  fusion 
in  -  target  recognition  and  surveillance,  2)  Sensor 
data  networks  across  imaging  and  non-imaging 
sensors. 


1  INTRODUCTION 

Imagery  and  Signal  intelligence  disciplines 
have  developed  to  high  levels  of  maturity  over 
the  past  40  years,  each  developing  mature 
technologies  to  collect,  analyze  and  exploit  data 
to  deliver  finished  intelligence  products  to 
users.  Within  each  of  the  disciplines,  fusion  of 
data  is  being  performed  to  enhance  the  overall 
intelligence  product,  and  to  maximize  the 
extraction  of  meaning  (intelligence)  from  the 
raw  data.  It  has  been  recognized  that  fusion 
between  the  intelligence  disciplines  offers  the 
potential  for  further  improvements  in  both 
performance  and  effectiveness. 

1 . 1  Fusion  Within  IMINT  and  SIGINT 

Representative  SIGINT  systems  with  highly 
automated  collection  and  processing  capabilities 
include  the  U.S.  RC-12  Guardrail,  RC-135  VAV 
Rivet  Joint,  or  EP-3  Aries  (see  ',  2).  These 
systems  employ  sophisticated  broadband 
receivers  to  collect  and  sort  signals,  both 
communications  and  more  general  electronic 
emissions  (radars,  jammers,  etc.)  to: 

•  Provide  efficient  scanning  of  multiple 
bands  to  achieve  high  intercept 
probabilities  for  intermittent  emitters, 
band-to-band  handoff,  instantaneous 
intercept,  record  and  “g  is  ting  ” 

(abstracting)  of  emitters. 
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•  Develop  an  electronic  order  of  battle 
describing  emitter  types,  quantities, 
functions,  relationships,  capabilities, 
and  tactical  behaviors. 

•  Locate  individual  emitters 

•  Monitor  use,  modes  and  capacity  of 
emitters  by  external  behavior,  or  by 
inference  from  internal  data  to  extract 
intelligence  data. 

Because  these  systems  search  multiple 
bands  for  emitters,  which  are  often  related, 
fusion  of  data  across  bands  must  be  performed  to 
develop  relationships  between,  for  example,  low- 
band  communications  and  high-band  radar 
activities.  SIGINT  testbeds,  such  as  the 
Canadian  Data  Fusion  and  Correlation 
Techniques  Testbed  (DFACTT) 3  are  developing 
and  refining  automated  techniques  to  correlate 
the  information  gleaned  from  multiple  SIGINT 
channels  to  support  analysts  operating  in  today’s 
dense,  complex  electromagnetic  environment. 

Similarly,  the  IMINT  community  has 
developed  automated  means  to  register  and 
combine  multiple  images  and  terrain  data  of  the 
same  ground  areas  to  enhance  the  utility  of 
imagery  intelligence  products.  Much  of  the  U.S. 
drive  to  digitize  and  visualize  the  battlefield  is 
based  upon  technologies  which  register  and 
combine  imagery,  maps,  digital  terrain  and  other 
spatial  data  sources  to  provide  a  comprehensive 
and  accurate  geographic  information  system 
(GIS)  to: 

•  Provide  efficient  use  of  archived  high- 
fidelity  imagery  and  tasking  of 
reconnaissance  imaging  sensors, 
sensor-to-sensor  cueing  and  handoff, 
near-real  time  collection,  exploitation 
and  extraction  of  objects  of  tactical 
interest. 

•  Develop  physical  orders  of  battle 
describing  identity  and  location  of 
fixed,  mobile  and  moving  objects  of 
tactical  interest. 


•  Develop  tracks  of  moving  targets  by 
revisits  at  sufficient  rates  to  maintain 
tracks 

•  Precisely  locate  individual  objects  for 
surveillance,  targeting,  and  change 
detection. 

•  Survey  targets  to  monitor  function  and 
behavior,  or  for  battle  damage 
assessment  (BDA). 

Next  generation  IMINT  systems,  such  as 
the  U.S.  Common  Imagery  Ground  Surface 
System  (CIGSS)  have  specified  common 
standards  for  imagery  formats  and  metadata4 
to  provide  the  capability  to  permit  spatial  data 
fusion  and  visualization  of  data  from  multiple 
IMINT  sources.5  Numerous  techniques  exist 
to  perform  imagery  and  spatial  data  fusion  to 
synthesize  two  and  three-dimensional  views 
of  the  battlefield  which  maximize  the 
information  content  from  all  available 
imaging  sensors.  Imagery  data  of  differing 
resolutions,  geospatial  accuracy  and  spectral 
content  may  be  registered,  orthorectified  to 
remove  distortion  due  to  terrain  relief, 
mosaicked  and  combined  to  form  a  common 
base  for  extracting  features,  generation  of 
rapid  map  updates,  and  analysis  for  a  variety 
of  military  applications.  (See  6  for  a  survey  of 
this  technology.) 

1.2  Issues  Facing  Intelligence  Fusion 
Between  the  IMINT  and  SIGINT  Disciplines 

Although  both  of  these  disciplines  have 
matured  and  are  applying  the  principles  of 
data  fusion  to  correlate,  merge  and  combine 
data,  the  challenge  of  fusion  IMINT-to- 
SIGINT  data  remains  a  challenge  due  to  the 
distinct  differences  in  SIGNT  and  IMINT  data 
sources.  Figure  1  illustrates  the  major 
difference  in  the  two  sources:  the  fidelity  of 
measurements  differ  in  each  of  the  two 
dimensions  of  observations.  EMINT  provides 
excellent  spatial  fidelity,  but  lacks  the  revisit 
rate  and  instantaneous  coverage  afforded  by 
SIGINT.  SIGINT  lacks  spatial  fidelity,  but 
provides  high  temporal  fidelity.  While  these 
are  general  characteristics,  they  apply  to  most 
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cases.  Consider  the  combination  of  UAV 
synthetic  aperture  radar  (SAR)  and  airborne 
electronic  support  (ES).  The  SAR  can  provide 
spatial  resolutions  on  the  order  of  one  meter 
or  better  (e.g.  spatial  accuracies  on  the  order 
of  meters  or  tens  of  meters)  and  revisits  of  a 
target  on  the  order  of  minutes  or  tens  of 
minutes  at  best.  A  tactical  ES  system  can 
provide  instantaneous  revisits  in  less  than  a 
second,  but  with  single-look  spatial  accuracies 
ranging  from  hundreds  of  meters  to 
kilometers. 
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Figure  1-  IMINT  -  SIGINT  Contributions 
provide  the  potential  for  high-fidelity  pace¬ 
time  intelligence  products. 

The  objective  of  fusing  IMINT  and 
SIGINT  data-  is  to  provide  combined 
intelligence  which  used  the  best  of  each 
source  -  a  combined,  high  fidelity  space-time 


product.  The  ability  to  achieve  this  fusion, 
however,  requires  a  means  to  unambiguously 
correlate  data  across  sources  to,  for  example, 
locate  a  specific  emitter  within  an  image. 

Figure  2  details  additional 
contrasting  characteristics  of  these  two 
sources: 

1)  MINT  captures  spatial  objects  (entities  or 
phenomena  from  events  -  pixels),  while, 

2)  SIGINT  captures  temporal  events 
(emission  events:  pulses,  modulations). 

3)  MIT  provides  inherent  spatial 
information  in  the  data,  in  contrast, 

4)  SIGINT  derives  spatial  information  from 
measurements  of  the  time  and  direction  of 
arrival  of  signals. 

5)  MINT  delivers  high-fidelity  spatial 
information  and  context,  while, 

6)  SIGINT  delivers  high-fidelity  temporal 
content  (and  derived  information,  e.g.  the 
identity  of  an  emitter,  or  internal  content) 
and  context. 

These  contrasting  characteristics  make 
the  two  sources  ideal  candidates  to  deliver 
high  fidelity  spatial  and  temporal  reports  on 
battlefield  targets  and  their  behavior.  These 
contrasting  characteristics  also  pose  major 
challenges  to  fusion: 

1)  The  platforms  and  collection  methods  of 
each  discipline  are  sufficiently  different 
that  coordination  of  sensors  to  achieve 
near-simultaneous  acquisition  of  both 


Characteristic 

IMINT 

SIGINT 
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EVENTS 

Information 

Content 

PRIMARY:  Sensed  data  describes 
real-world  entities  and  their  spatial 
context 

SECONDARY:  Time  of  observation 

PRIMARY:  Sensed  data  describes 
electronic  events  (emissions)  and 
their  temporal  context  (waveforms) 
SECONDARY:  Location  derived  from 
direction  and  time  of  observation 

Context  of  Data 

SPATIAL 

TEMPORAL 
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Time  of  occurrence 
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Physical  Objects 
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Focus  of 

Analysis 

Signature  of  spatial  shape,  size, 
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Figure  2  -  Contrasting  Features  of  IMINT  and  SIGINT  Sources 
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EMINT  and  SIGINT  data  is  difficult. 

2)  Spatial  correlation  between  IMINT 
objects  (entities  in  a  scene)  and  SIGINT 
objects  (spatial  ellipses)  provides 
ambiguity  and  uncertainty  due  to  the  large 
difference  in  spatial  resolutions  of  the 
data  sets,  and  the  time  difference  in 
collection. 

3)  Combination  of  the  data  is  also  difficult, 
because  the  sources  each  reports  different 
characteristics  about  the  entities  and 
events. 

1.3  Benefits  of  IMINT-SIGINT  Fusion 

The  expected  benefits  of  automated 
fusion  of  IMINT  and  SIGINT  data  include: 

•  Improved  tasking  and  utilization 
efficiency  of  both  SIGINT  and  IMINT 
sensors,  via  improved  sensor-to-target 
pairing. 

•  More  rapid  warning,  cueing  and 
intelligence  understanding  in  the  presence 
of  uncertainty. 

•  Improved  detection,  identification, 
characterization  and  geolocation  of  targets 
of  interest.  (The  improvement  may 
provide  a  reduction  in  uncertainty, 
resolution  of  conflicts,  improvement  in 
accuracy.) 

•  Rapid  geolocation  of  threats  for  sensor-to- 
shooter  targeting.  7  (This  improvement 
may  include  a  reduction  in  time-to-locate 
with  sufficient  precision  to  react.) 

These  contributions  are  important 
technical  elements  needed  to  achieve  the 
information  superiority  objectives  on  the 
modem  battlefield.  The  U.S.  Advanced 
Battlespace  Information  System  (ABIS)  Task 
Force  identified  multisensor  fusion  and  sensor 
cross-cueing  as  a  “needed  technology”  to 
achieve  precision  information  direction.8  The 
task  force  identified  the  progressive 
incorporation  and  integration  of  IMINT  and 
SIGINT  sensors  as  an  ultimate  stage  to 
achieve  automatic  sensor-to-target  pairing 
capabilities.9  One  example  of  the  move 
toward  these  capabilities  is  the  consideration 


of  developing  Global  Hawk  UAV  payloads 

that  will  autonomously  cue  imaging  sensors 

by  emissions  detected  by  Electronic  Support 
10 

sensors. 

Traditionally,  IMINT  and  SIGINT  are 
combined  by  intelligence  analysts  who  have 
oversight  and  access  to  both  imagery  and 
signals  data.  With  broad  intelligence 
oversight,  their  manual  fusion  functions 
include  the  following  kinds  of  actions: 

•  Overlay  SIGINT  data  (error  ellipses, 
lines  of  bearing)  on  archive  imagery 
or  maps,  and  search  recent  imagery 
for  signs  of  the  physical  emitter. 

•  Cue  IMINT  sensors  to  look  in  regions 
of  emitter  activities. 

•  Cue  SIGINT  sensors  to  search  for 
emissions  in  areas  where  military 
activity  is  located  in  imagery. 

Automation  of  these  types  of  IMINT- 
SIGINT  fusion  functions  requires  the 
ability  to  automate  the  following 
activities: 

Alignment  ( Registration )  -  Spatial  and 

temporal  alignment  of  the  two  sources 
must  be  performed  to  relate  the  data  sets. 
Spatial  alignment  requires  transformation 
of  the  signal  location  data  onto  imagery 
coordinates;  this  generally  requires  the 
mapping  of  an  uncertainty  volume  (e.g. 
error  ellipse)  onto  an  image  surface. 

Correlation  -  Possible  associations 
between  SIGINT  objects  (e.g.  an  error 
ellipse)  and  IMINT  object  (e.g.  all 
military-like  entities  within  the  ellipse) 
are  enumerated,  and  an  association  metric 
is  computed  for  each.  The  metric  may 
account  for  spatial,  temporal,  behavioral 
and  identifying  feature  similarities.  A 
single,  highest  scoring  candidate  may  be 
assigned  (“correlated”)  or  multiple 
hypotheses  may  be  correlated  and 
retained  for  further  consideration  and 
refinement  as  more  data  arrives 
(“deferred-decision”  or  “multiple 
hypothesis”). 
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Combination  -  Correlated  objects  may  be 
combined  in  various  ways  to  refine  or 
verify  the  identification  of  the  target 
entity.  If  the  emitter  is  a  general  fire 
control  director,  for  example,  and  the 
correlated  objects  in  imagery  reveal  a 
particular  firing  unit,  the  specific  category 
of  missile  system  may  be  identified. 

Reasoning  -  Contextual  data  (e.g.  terrain, 
road  networks,  lines  of  communications 
and  line-of-sight  considerations)  may 
also  be  used  to  support  the  correlation  and 
combination  processes  to  restrict 
alternatives. 

Resource  Tasking  -  Tasking  of  collection 
and  processing  resources  may  be 
coordinated  to  provide  simultaneous,  near 
simultaneous  or  synchronized  acquisition 
of  data  and  efficient  processing. 


In  the  following  sections,  we  discuss 
the  functional  opportunities  to 
integrate  and  automate  IMINT  and 
SIGINT  sources  and  the  technologies 
that  will  permit  this  automation. 

2.  ALTERNATIVE  METHODS  FOR 
IMINT-SIGINT  FUSION 

A  number  of  alternatives  exist  to  improve 
the  coordination  and  combination  of  IMINT 
and  SIGINT  Sources.  Figure  3  illustrates  the 
typical  independent  intelligence  collection 
and  processing  “stovepipes”  and  the  four 
major  areas  where  improvements  can  be 
achieved.  The  following  four  subsections 
detail  each  of  the  four  areas  of  opportunity 
and  offer  conceptual  approaches  to  perform 
fusion  functions. 
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Figure  3-  Four  major  areas  of  coordination  and  combination  exist 
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2.1  Coordinate  Tasking,  Cueing  &  Control 

The  coordination  of  tasking,  and  real-time 
cross  cueing  of  sensors  hold  the  promise  to 
improve  the  combined  coverage  volume,  and 
speed  to  focus  on  targets  of  interest.  Several 
tasking  modes  may  be  considered  (Table  1) 
provide  the  ability  to  perform  combined 
search  and  iterative  focus  to  improve  the 
composite  target  intercept  performance  (for 
SIGINT)  and  characterization  (for  IMINT). 


Table  1  -  Candidate  Coordinated 
Tasking  and  Cross  -Cueing  Modes 


Mode 

Functions 

SIGINT 
Area  Search 

•  Search  broad  areas  for 
emitters  of  interest 

•  Cue  IMINT  to  locate  and 
verify  detected  targets 

IMINT 

Area 

Search 

•  Search  broad  areas  for 
non-emitting  mobile  and 
fixed  targets 

•  Cue  SIGINT  to  focus  on 
probable  emitting  tarqets 

SIGINT 

Focus 

•  Task  dwell  and  precise 
tune  on  designated  area  to 
verify  identity  and  location 

IMINT 

Focus 

•  Task  precise  imagery,  or 
image  sequence  on 
designated  area  to  verify 
identity  and  location 

Consider  the  following  representative 
coordination  sequence  between  the  two 
sources  (as  indicated  by  the  numerical 
sequence  in  figure  4): 

1.  SIGINT  are  search  locates  a  low 
probability  detection,  with  very  coarse 
error  ellipse; 

2.  A  cross  cue  to  IMINT  is  generated, 
requesting  a  search  of  the  area; 

3.  EMINT  searches  archive  imagery  for 
potential  locations,  and  tasks  the  next 
available  collector  to  collect  imagery 
over  the  probable  locations; 

4.  The  probable  locations  are  used  to  cue 
SIGINT; 

5.  SIGINT  collects  high  dwell  and 
precisely  tuned  data  over  the  probable 
areas,  then  issues  reports  to  IMINT; 


6.  IMINT  used  this  data  to  search 
recent  imagery  (collected  by  the 
tasking  in  step,  3,  earlier)  to 
detect  and  locate  the  target. 

The  state  diagram  in  Figure  4  illustrates 
the  iterative  and  supportive  sequence  by 
which  the  two  sources  can  be  tasked  and  cued 
to  refine  low-confidence  detections  to  either 
eliminate,  or  validate  and  locate  targets. 


Figure  4-  Transition  of  Search  and  Focus 
States  for  Coordinated  cueing  and  tasking 


This  kind  of  cooperative  and  iterative 
search  requires  very  close  coordination  of 
both  sources,  real-time  interaction  between 
processing  of  both  data,  and  the  ability  to 
rapidly  re-task  both  sensors.  The  U.S. 
Integrated  Collection  Management  ACTD 
(Advanced  Concept  Technology  Demo)  is 
developing  these  kinds  of  capabilities,  and  has 
published  a  desired  goal  of  2-4  hours  for 
response  time  between  sources  to  achieve 
“tasking  within  friendly  and  enemy  cycle 
times.”11 

2.2  Fusion  for  Improved  Geolocation 

The  coordination  process  described 
above  can  yield  improvements  in  the  geo¬ 
location  of  critical  targets  that  have  been 
located  within  coarse  SIGINT  error  ellipses. 
The  use  of  imagery  and  terrain  data  to  search 
probable  locations  (and  to  delimit  unlikely 
locations)  within  the  coarse  SIGINT  ellipse 
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may  yield  precise  locations  or  precise 
probable  locations. 

2.3  Fusion  for  Increased  Target  Detection 
and  Identification  Confidence 

In  addition  to  geolocation  improvement, 
the  process  may  also  provide  for  more 
accurate  detection  and  identification  of 
targets.  Consider  the  following  possible 
situations  in  which  joint  IMINT  and  SIGINT 
data  may  be  combined  to  improve  detection 
and/or  identification: 

1)  SIGINT  may  locate  an  individual  emitter, 
but  an  IMINT  search  refines  the 
associated  military  components  associated 
with  the  emitter. 

2)  Low-confidence  SIGINT  hits  may  be 
rejected  or  upgraded  to  higher  confidence 
with  collaborating  IMINT  or  terrain 
contextual  knowledge. 

3)  IMINT  data  may  identify  candidate 
activity  areas,  with  no  observable  targets, 
and  SIGINT  may  be  tasked  to  search  for 
relevant  emissions  to  confirm  or  eliminate 
the  areas. 

4)  SIGINT  data  may  precisely  locate  an 
emitter,  with  ambiguous  identity,  and 
IMINT  may  provide  contextual  data  to 
support  high-confidence  identification. 

Programs  such  as  the  U.S.  DARPA 
Dynamic  Multi-user  Information  Fusion 
(DMIF)  program  are  developing  state-of-the- 
art  methods  to  combine  IMINT  and  SIGINT 
report  objects.  In  that  program,  tactical 
decision  reports  (messages  containing  identity 
and  location  of  targets)  are  parsed,  common 
objects  are  correlated,  and  linkages  between 
objects  are  discovered  using  force  level 
structure  and  behavioral  templates. 12 

2.4  Improved  Visualization  of  the  Battlefield 

The  effective  coordinated  tasking,  correlation 
and  combination  of  IMINT  and  SIGINT 
sources  and  data  provide  the  potential  to 
maintain  comprehensive  views  of  the 
battlefield  -  and  to  present  these  in  dynamic 
visual  presentations  to  optimize  human 
comprehension  of  the  battle  situation. 


Visualizations  of  joint  IMINT  and  SIGINT 
data  may  take  on  many  forms,  dependent 
upon  system  and  user  requirements,  but 
several  alternative  candidates  are  presented  in 
Figure  5.  The  figure  illustrates  three 
successive  levels  of  sophistication  in 
overlaying  a  coarse  SIGINT  error  ellipse  on 
imagery-derived  products  to  aid  an  analyst  in 
searching  for  potential  sources.  These 
visualizations  provide  both  the  temporal 
context  of  SIGINT  and  the  spatial  context  of 
IMINT  in  a  common  form  -  and  should  allow 
the  analyst  to  “drill-down  “  into  the  data  bases 
from  either  source  to  improve  the 
understanding  of  the  battlefield. 


APPROACH  EXAMPLE  VISUAL 


SIGINT 
Overlay 
on  3-D 
Imagery 
Visual 


SIGINT 
Overlay  on 
3-D  Visual 
with 

Annotation 


Figure  5-  Alternative  combined 
visualizations  of  IMINT-SIGINT 
data. 
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4.  CONCEPTUAL  FUTURE  SOLUTIONS 

The  ability  to  achieve  effective  MINT- 
SIGINT  fusion  requires  the  integration  of 
several  technologies  and  system 
commitments;  the  U.S.  programs  cited 
throughout  this  paper  have  illustrated  some  of 
the  efforts  to  accomplish  this.  The 
technologies  and  system  commitments  to 
make  this  possible  include: 

•  Refined  Collection  Assets  -  The 
introduction  of  medium  and  high  altitude 
endurance  unmanned  air  vehicle 
(UAV’s),  in  concert  with  manned 
platforms,  will  provide  more  continuous, 
longer  dwell  views  by  MINT  to  increase 
the  potential  for  near-simultaneous 
collection  with  SIGINT.  This  will 
increase  the  effectiveness  of  cross-cueing 
for  dynamic  targets,  while  increasing  the 
likelihood  that  MINT  can  be  tasked  to 
geolocate  SIGINT  targets. 

•  Coordinated  Intelligence  Collection 
Architectures-  The  MINT  and  SIGINT 
information  chains  cannot  be  operated  as 
independent  “stovepipes”  to  achieve 
timely  coordination  of  tasking  and 
exploitation.  The  architecture  for 
planning,  tasking  and  exploitation  must 
provide  for  rapid  interaction  based  on 
overall  prioritization  of  joint  resources. 
As  an  example,  the  U.S.  Defense 
Airborne  Reconnaissance  Office 
(DARO),  has  defined  a  “Technical 
Architecture”  which  provides  data  paths 
between  MINT  and  SIGINT  with  the 
requirements  for:  l)“reach-back”  (an 
operation  where  sensor  data  is  passed 
back  to  supporting  site  facilities  for 
processing  and/or  exploitation/analysis) 
and,  2)  reach-forward”  (capabilities  for 
data  from  supporting  intelligence  centers 
to  be  forwarded  to  the  airborne 
reconnaissance  system  for  relay, 
exploitation,  cross-system  cueing,  data 
correlation,  or  other  intelligence 
functions )  capabilities.13 

•  Source  Data  Reporting-  SIGINT  and 
MINT  report  formats  (and  data  content) 
should  provide  metrics  to  support  the 
correlation  and  combination  processes. 


These  metrics  include:  time  of  collection, 
common  statistical  uncertainty  measures 
for  location  and  identification, 
exploitation  support  data,  etc. 

•  Exploitation  Architectures  and 
Processing  -  The  exploitation 
architectures  must  also  provide  the 
capability  to  store,  link  and  jointly 
visualize  both  sources  of  data.  Fusion 
algorithms  providing  correlation  and 
combination  (managing  uncertainty  and 
able  to  provide  multiple  hypotheses  or 
uncertainty  metrics  to  users)  is  also 
required. 

•  Precision  Terrain  and  Feature  Data  - 
Spatial  context  data  consisting  of 
precision  terrain,  feature  and  other  data 
(See  14  for  description  of  U.S.  National 
Imagery  and  Mapping  Agency’s 
categories  of  topographic  data,  and  15 
and  16  for  a  typical  analyses  of  the 
applicability  of  this  data  for  military 
application.) 

In  addition  to  technology  changes, 
doctrinal  changes  will  also  be  required  to 
exploit  the  potential  of  MINT-SIGINT 
fusion.  In  recent  test  exercises,  such  as  the 
massive  Force  XXI  Advanced  Warfighting 
Experiment  (AWE),  the  Army’s  1st  Brigade 
Experimental  Force  (EXFOR)  reported 
significant  benefits  of  fusing  and  cross-cueing 
UAV  MINT  with  ground  and  airborne 
SIGINT.17  But  the  exercise  also  illustrated 
the  need  of  rethinking  operational  procedures 
to  complements  the  introduction  of  fusion 
technologies.  The  kinds  of  doctrinal  changes 
that  may  be  considered  include: 

•  Joint  Sensor  Utilization  -  As  the 
technology  to  allow  coordinated  tasking 
is  developed,  so  must  the  doctrine  to 
support  this  new  use  of  the  assets  -  as  a 
joint  sensor  network,  rather  than  two 
independent,  but  cooperating,  sources. 

•  Fusion  Analyst  Position-  Analysts 
with  joint  MINT  and  SIGINT  skills  may 
be  required  to  take  maximum  benefit  of 
integrated  visualizations.  A  new 
intelligence  discipline,  the  fusion  analyst, 
may  be  required,  with  an  appropriate 


training  curriculum,  to  reap  the  full 
potential. 

4.  CONCLUSION 

The  automated  fusion  of  imagery  and  signal 
intelligence  data,  now  performed  manually 
and  infrequently,  has  the  potential  to  provide 
significant  timeliness,  and  information 
accuracy  benefits  to  the  warfighter.  In  order  to 
achieve  the  maximum  benefits  of  automation, 
1)  system  architectures  must  be  structured  to 
permit  collection  interaction  and  coordination 
of  the  intelligence  chains,  2)  fusion  processing 
and  visualization  technologies  must  be 
applied  to  optimize  the  extraction  of  joint 
intelligence,  and,  3)  supporting  data,  such  as 
precision  terrain  data,  must  be  available  to 
permit  contextual  reasoning  about  the  earth 
terrain. 
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PAPER  No:  35 

DISCUSSOR’S  NAME:  G.  Wyman 

COMMENT/QUESTION: 

Can  you  identify  a  priori  the  domain  in  which  you  will  undertake  the  correlation? 


AUTHOR/PRESENTER’S  REPLY: 

The  domain  of  the  correlation  operation  can  be  described  by  the  following  example  sequence: 

1 .  SIGINT  detects  a  new  threat  emitter,  Type  X. 

2.  The  SIGINT  uncertainty  ellipse  area  is  detrermined  and  the  EMINT  sensor  (e.g.  a  SAR  on 
an  unmanned  air  vehicle)  is  tasked  to  collect  imagery  for  the  area). 

3.  The  emitter  type  (X)  is  used  to  infer  the  types  of  equipment  (vans,  shelters,  antennas) 
expected.  These  are  used  to  form  a  matched  filter. 

4.  The  matched  filter  is  scanned  across  the  imagery  (automatic  target  recognition)  to  locate 
the  equipment  associated  with  emitter  X  in  the  imagery. 

The  correlation,  therefore,  is  performed  in  the  image  domain,  using  a  reference  template  inferred 

from  the  SIGINT  detection. 


DISCUSSOR’S  NAME:  A.  Hume 

COMMENT/QUESTION: 

How  does  the  proposed  system  cope  with  multiple  emitters  in  the  imager’s  view? 


AUTHOR/PRESENTER’S  REPLY: 

The  concept  is  focused  on  locating  individual,  high-value  emitters  within  imagery.  While  many 
emitters  are  detected  across  the  battlefields,  the  SIGINT  system  will  select  high-value,  time-critical 
targets  (e.g.  mobile  SAM’s)  and  will  cue  IMINT  to  locate  those  individual  targets. 
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PAPER  NO:  35 

DISCUSSOR’S  NAME:  G.S.  Brown 

COMMENT/QUESTION: 

Does  the  use  of  spaceborne  imaging  assets  have  any  impact  on  your  fusion  which  was  primarily 
aimed  at  airborne  platforms  use? 

AUTHOR/PRESENTER’S  REPLY: 

Airborne  high-altitude  endurance  (HAE)  surveillance  platforms  (.e.g.  Global  Hawk  and  Darkstar) 
provide  a  capability  for  rapid  revisit  and  immediate  tasking  that  is  not  feasible  with  spaceborne 
assets,  such  as  SPOT  or  Radarsat. 


NOSTRAMARINE  :  UN  CONCEPT  DE  DETECTION  MULTISTATIQUE 
ADAPTE  A  LA  SURVEILLANCE  DES  CIBLES  BASSE  ALTITUDE. 
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RESUME 

Ce  papier  decrit  un  nouveau  concept  de 
detection  transhorizon  adapte  a  la  surveillance  des 
cibles  basse  altitude.  Le  radar,  de  configuration 
bistatique,  est  constitue  d'un  systeme  de  reception 
implante  au  voisinage  de  la  zone  a  surveiller,  et  d'un 
emetteur  eloigne,  situe  en  retrait  de  la  zone  a 
surveiller.  Le  fonctionnement  global  du  radar 
repose  sur  l'association  d'un  mode  de  propagation 
'ionospherique  ’  et  d'un  mode  de  propagation  par 
'ondes  de  surface'. 

Ainsi,  grace  aux  proprietes  de  reflexion  des 
ondes  electromagnetiques  sur  l'ionosphere,  les 
cibles  basse  altitude  peuvent  etre  illuminees  par 
l'emetteur  eloigne,  suppose  fonctionner  en  hautes 
frequences.  Les  ondes  electromagnetiques,  apres 
interaction  avec  les  cibles  se  propagent  en  suivant 
la  courbure  terrestre  (  par  'ondes  de  surface') 
jusqu'au  reseau  de  reception.  Ce  concept  constitue 
une  alternative  au  radar  transhorizon  par  ondes  de 
surface  traditionnel  (OTH-S),  en  presentant  au 
niveau  de  la  zone  geographique  a  surveiller, 
1'avantage  d'un  dispositif  passif,  discret,  et  dote  de 
capacites  d'integration  interessantes. 

La  propagation  joue  un  role  determinant  dans 
l'optimisation  des  caracteristiques  de  ce  dispositif; 
aussi  doit  on  mener  un  choix  judicieux  de  la 
frequence,  en  fonction  des  previsions 
ionospheriques  d'une  part,  et  des  conditions 
d'excitation  des  ondes  de  surface  d'autre  part. 

Du  point  de  vue  systeme,  remission  peut 
provenir  d'un  emetteur  radar  (par  exemple  OTH 
'Nostradamus')  ou  d'un  emetteur  de  radiodiffusion 
puissant.  Le  fonctionnement  en  reception  repose  sur 
une  antenne  reseau  a  formation  de  faisceaux  par  le 
calcul  a  laquelle  sont  appliquees  des  techniques  de 
filtrage  adaptatif.  Enfin,  la  fusion,  en  reception,  des 
plots  de  detection  obtenus  a  partir  d’emetteurs  de 
nature  et  emplacement  differents  ( multistatisrne  ) 
conlere  au  dispositif  des  proprietes  interessantes  en 
matiere  de  detection  et  de  localisation.  Des  resultats 
de  simulation,  illustrant  les  capacites  de  detection  et 
l'interet  d'associer  les  plots  issus  de  plusieurs 
emetteurs  eloigncs,  sont  decrits.  Ce  concept  de 
detection  a  conduit  a  des  campagnes  de  mesures 
sur  cibles  reelles;  un  resultat  obtenu  a  partir  d’un 


emetteur  de  radiodiffusion  non  cooperatif  est 
presente. 

INTRODUCTION 

La  detection  des  cibles  rasantes,  depuis  un  site 
cotier  ou  un  batiment  de  surface  pose  un  probleme 
de  preavis  d'alerte  pour  les  radars  classiques  dont  la 
visibility  'quasi  optique'  permet  tout  juste  de 
detecter  les  cibles  a  leur  lever  sur  l’horizon.  A 
moins  de  disposer  le  radar  sur  un  point  haut  -  ce  qui 
n'est  pas  toujours  envisageable  -  la  portee  basse 
altitude  se  trouve  limitee  a  quelques  dizaines  de 
kilometres,  ce  qui  est  faible  compte  tenu  des 
vitesses  potentielles  des  cibles  et  du  temps  de 
reponse  limite  des  procedures  de  riposte  a 
declencher  en  cas  d'alerte. 

Les  concepts  de  radars  transhorizon 
fonctionnant  en  bande  decametrique  (radar  OTH-B 
et  radar  OTH-S)  sont  deja  anciens.  Solution  au 
probleme  de  la  detection  des  cibles  basse  altitude 
(malgre  une  precision  de  mesure  nettement  en 
retrait  par  rapport  ce  que  fournissent  les  dispositifs 
hyperfrequences  traditionnels)  ce  type  de  radar 
semble  toujours  presenter  un  interet.  Les  progres  en 
matiere  de  traitement  numerique  du  signal  ont 
perrnis  d'accedcr  aux  performances  optimales  de  ces 
dispositifs  dont  le  fonctionnement  est  largement 
base  sur  la  formation  de  faisceaux  par  le  calcul 
(FFC)  et  le  traitement  Doppler  coherent.  D’autre 
part,  la  difficulty,  en  gamme  HF  (mais  egalement  en 
V-UHF)  d’utiliser  des  materiaux  absorbants 
capables  de  diminuer  la  reflectivity  des  cibles 
renforce  l’interet  porte  de  maniere  generate  aux 
dispositifs  fonctionnant  en  basse  frequence, 
dispositifs  que  Ton  peut  qualifier  «  d’antifurtifs  ». 

Le  dispositif  radar  decrit  dans  ce  papier 
appartient  a  la  classe  des  radars  transhorizon  bases 
au  sol  et  operant  en  gamme  HF.  De  configuration 
bistatique,  il  exploite  d'une  part  remission  d'une 
onde  de  ciel  dirigee  vers  les  cibles  a  detecter 
(emission  en  provenance  d'un  radar  OTH  ou  d'un 
emetteur  HF  de  forte  puissance),  il  s'attache  d'autre 
part  a  traiter  de  maniere  optimale  les  ondes  de 
surface  induites  par  les  cibles  evoluant  au  voisinage 
de  la  surface.  Le  concept  exploite  les  modes  de 
propagation  bien  connus  en  HF  et  notamment 
utilises  dans  le  radar  transhorizon  par  onde  de  ciel 
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(OTH-B)  et  le  radar  transhorizon  par  ondes  de 
surface  (OTH-S).  L’appellation  ‘Nostramarine’ 
resulte  d'une  contraction  de  "Nostradamus"  (radar 
transhorizon  developpe  par  l'Onera  pour  le  compte 
du  SPAe  a  Dreux  Senonches)  et  du  "Radar  HF 
marine  (projet  de  radar  a  ondes  de  surface  etudie 
par  l'Onera  pour  de  compte  de  la  DCN  en  1 989)". 

Le  principe  de  fonctionnenient  du 
dispositif  Nostramarine  est  decrit  dans  la  premiere 
partie  de  ce  papier.  La  seconde  partie  traite  des 
performances  theoriques;  des  resultats  de 
modelisation  de  la  couverture  basse  altitude  sont 
notamment  presentes.  Le  traiternent  du  signal  est 
ensuite  decrit.  Enfin,  dans  la  derniere  partie,  un 
resultat  experimental,  obtenu  a  partir  d'un  emetteur 
d'opportunite,  est  presente. 

1.  PRINCIPE  DE  FONCTIONNEMENT 

1.1.  Les  radars  transhorizon  existants 

Les  radars  transhorizon  fonctionnant  en 
gamme  decametrique  sont  de  deux  types: 

-  les  radars  transhorizon  par  reflexion 
ionospheriques  (OTH-B)  dont  les  portees  de 
detection  peuvent  atteindre  plusieurs  milliers  de 
kilometres.  Ces  radars  possedent  une  zone  de 
silence  ("zone  aveugle")  de  l’ordre  de  500  km  a  750 
km. 

-  les  radars  a  ondes  de  surface  (OTH-S), 
fonctionnant  exclusivement  au  dessus  de  la  mer 
avec  des  portees  de  detection  pouvant  atteindre 
plusieurs  centaines  de  kilometres. 

Les  radars  OTH-B  sont  employes  pour  des 
missions  de  surveillance  aerienne  tres  longue 
portee,  egalement  pour  des  missions  de  surveillance 
antibalistique.  Les  radars  OTH-S  sont  employes 
pour  des  missions  de  surveillance  transhorizon  au 
dessus  de  la  mer:  surveillance  vis  a  vis  des  missiles 


en  penetration  basse  altitude.  egalement 
surveillance  des  bateaux(lutle  contre  le  passage  de 
la  drogue,..).  L'utilisation  de  tels  dispositifs  a  bord 
de  batiments  dc  surface  est  egalement  envisagee 
pour  contrer  la  menace  des  missiles  mer-mer  et  pour 
detecter  de  maniere  precoce  la  presence  dautres 
batiments  de  surface.  En  revanche  I'importance  des 
niveaux  de  puissance  a  emettre  ainsi  que  la 
dimension  des  aeriens  d’emission  constituent  un 
point  dur  a  l'emploi  d'un  tel  radar  a  bord  de 
batiments  de  surface. 

1.2.  Le  concept  "passif"  Nostramarine 

Le  concept  "Nostramarine"  repose  sur  la 
mise  en  oeuvre,  en  association  avec  un  emetteur 
transhorizon,  d'un  reseau  de  reception  au  voisinage 
de  la  zone  de  menaces  a  surveiller.  La  figure  1 
decrit  schematiquement  le  principe  de 
fonctionnement  du  radar.  Dans  la  zone  a  surveiller. 
evoluent  des  cibles  qui  sont  eclairees  par  les  ondes 
de  del  provenant  d’un  emetteur  de  forte  puissance 
(radar  transhorizon  ou  dmetteur  de  radiodiffusion 
puissant).  Les  cibles  diffusent  I'encrgic 
electromagnetique  dans  toutes  les  directions:  si  les 
cibles  evoluent  a  basse  altitude  cette  diffusion 
genere  par  couplage  avec  la  mer  des  ondes  de 
surfaces  qui  se  propagent  en  suivant  la  rotondite 
terrestre,  conferant  au  dispositif  une  capacite  de 
detection  transhorizon.  Les  ondes  de  ciel  provenant 
de  l'emetteur  parviennent  au  voisinage  de  la  cible 
avec  une  polarisation  generalement  elliptique; 
l'interaction  des  ondes  incidentes  avec  la  cible  est 
complexe;  en  revanche  settles  les  ondes  polarisees 
verticalement  se  propagent  le  long  de  la  surface  de 
la  mer  vers  le  recepteur.  Le  reseau  de  reception  pent 
etre  implante  en  bordure  de  cote  (surveillance 
cohere)  ou  a  bord  d'un  batiment  de  surface. 


IONOSPHERE 


fig.  1  :  principe  de  fonctionnement  du  dispositif  Nostramarine 
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reception 


fig.2  :  separation  des  signaux  en  reception 


Le  role  du  reseau  d'antennes  est  de  separer  et 
selectionner  les  ondes  de  ciel  (trajet  direct) 
provenant  directemcnt  de  l'emetteur  (1)  des  ondes 
de  surfaces  provenant  des  cibles  (2).  La  selection 
des  differentes  ondes  est  faite  par  traitement 
adaptatif,  les  antennes  du  reseau  etant  couplees  a 
une  batterie  de  recepteurs  et  de  codeurs  permettant 
de  proceder  au  traitement  numerique  des  signaux. 
Le  traitement,  dont  les  bases  seront  detaillees  par  la 
suite  consiste  a  former  une  voie  de  reference  (3) 
dans  la  direction  du  trajet  direct  et  une  voie  utile  (4) 
dans  la  direction  presumee  des  cibles. 

Le  fonctionnement  du  radar  n'est  effectif 
qu'apres  une  phase  de  calibration  a  remission  et  a  la 
reception.  La  calibration  a  remission  comprend 
plusieurs  operations;  dans  le  cas  d'un 
fonctionnement  couple  a  un  radar  transhorizon,  ces 
operations  sont,  chronologiquement : 

-  le  choix  de  la  gamme  de  frequence  :  le  choix 
grossier  de  la  frequence  (a  1  MHz  pres)  est  effectue 
a  l'issue  d'une  prevision  des  conditions  de 
propagation  ionospherique.  Le  choix  de  cette 
frequence  est  ensuite  affine  de  maniere  a 
positionner  precisement  le  faisceau  d’emission  sur 
la  zone  a  surveiller  ( pilotage  du  faisceau 
d’ emission)-,  cette  affinage  conduit  a  determiner  la 
frequence  avec  une  marge  de  l’ordre  de  100  a  200 
KHz.  A  l’interieur  de  cette  bande  de  frequence,  on 
recherche  alors  une  frequence  claire  presentant  le 
niveau  de  bruit  le  plus  faible  ( recherche  de 
frequence  claire). 

-  le  pilotage  du  faisceau  d'emission:  dans  le  cas  de 
1’ utilisation  d'un  emetteur  radar,  au  faisceau  directif, 
une  procedure  de  positionnement  du  faisceau  est 
requise.  Cette  procedure  peut  etre  menee  de  fagon 
entierement  adaptative,  grace  a  la  conjugaison  de 
phase.  Dans  ce  cas,  on  dispose  au  niveau  du  reseau 
de  reception  (site  local)  un  petit  systeme  d'emission 
de  faible  puissance  (inferieure  a  50  w),  fonctionnant 
a  une  frequence  F0.  Le  radar  OTH  regoit  sur  chaque 
antenne,  le  signal  a  la  frequence  F0  et  extrait  apres 


demodulation  le  vecteur  directionnel  des  phases  du 
signal  regu.  Ce  vecteur  directionnel  signe 
l'emplacement  du  reseau  de  reception  Nostramarine 
et  est  alors  utilise  en  emission  pour  focaliser 
l'energie  du  radar  dans  la  direction  du  reseau  local. 
La  conjugaison  de  phase  permet  de  compenser  les 
erreurs  de  phases  ionospheriques  et  garantit  ainsi 
une  focalisation  optimale  du  faisceau  d’emission. 
La  liaison  necessitee  par  la  calibration  en  emission 
doit  se  faire  sur  une  frequence  F0  claire  au  niveau 
du  radar  OTH.  L'emission  du  radar  OTH  vers  le 
reseau  local  peut  se  faire  a  une  frequence  F| 
differente.  Cette  solution  est  preferable:  en  effet, 
d'une  part  une  recherche  de  frequence  claire  unique 
et  valable  sur  les  deux  sites  est  contraignante, 
d'autre  part  le  changement  de  frequence  permet  de 
deplacer  la  tache  focalisee  au  sol  le  long  du  segment 
OTH-Reseau  local;  ceci  permet  d’ajuster  au  mieux 
l’emplacement  du  faisceau  d’emission  sur  la  zone  a 
surveiller.  En  augmentant  legerement  la  frequence 
(Fj  >  F0),  la  tache  au  sol  s’eloigne  du  site  de 
reception.  Cette  situation  schematisee  par  la  figure 
3  est  interessante  dans  un  contexte  d'alerte  cohere. 


fig.  3  :  pilotage  du  faisceau  d'emission 


-  la  recherche  de  frequence  claire  :  effectuee  au 
niveau  du  site  de  reception,  elle  consiste  a  analyser 
des  canaux  de  largeur  comprise  entre  5  et  30  KHz 
(selon  la  largeur  de  bande  requise  par  remission 
radar)  dans  la  gamme  de  frequence  determinee 
apres  calibration  du  faisceau  d’emission.  La 
recherche  de  frequence  claire  doit  etre  frequemment 
reactualisee  car  l’environnement  radioelectrique 
fluctue  de  maniere  importante  en  gamme  HF.  Le 
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recours  a  une  procedure  de  recherche  de  frequence 
claire  simultanee  (ou  entrelacee)  avec  le 
fonctionnement  du  radar  constitue  la  solution 
ideale. 

La  calibration  en  reception  est  effectuee  a  partir  du 
trajet  direct  en  exploitant  la  connaissance  a  priori  de 
la  position  de  l’emetteur  utilise  et  les  previsions 
ionospheriques  des  conditions  de  propagation. 

1.3.  Avantages  et  inconvenients  du  dispositif 
Nostramarine 

Le  reseau  deploye  au  voisinage  du  secteur 
a  surveiller  est  entierement  passif.  Ce  dernier  est  par 
consequent  tres  peu  vulnerable  ,  notammcnt  vis  a 
vis  des  missiles  antiradar.  Les  antennes  peuvent 
etre  de  dimensions  reduites;  l'utilisation  de  fouets 
actifs  large  bande  (hauteur  =  1  a  2  m)  permet  de 
concevoir  un  reseau  facilement  deployable  ne 
necessitant  aucune  infrastructure.  Rappelons,  qu'a 
l'inverse,  un  radar  OTH  monostatique  utilisant  des 
antennes  d'emission  accordees,  presente  des 
dimensions  importantes  (antenne  log-periodique  ou 
antennes  biconiques  de  hauteur  superieure  a  10  m). 
Enfin,  l'implantation  d'antennes  de  reception  de 
petites  dimensions  est  envisageable  sur  un  batiment 
de  surface  (fregate..).  Par  ailleurs,  le  risque  de 
perturbations  radioelcctriques  causees  par  le  radar 
sur  les  equipements  du  batiment  est  limite. 

Enfin,  contrairement  aux  radars  OTH-S 
monostatiques,  la  configuration  bistatique  de 
Nostramarine  lui  permet  de  fonctionner  avec  des 
formes  d'onde  continues,  et  sans  distance  aveugle. 
(la  distance  aveugle,  dans  le  cas  de  radar  OTH-S 
monostatique  est  de  l'ordre  de  20  km.) 

Le  premier  inconvenient  du  dispositif 

Nostramarine,  si  on  le  compare  a  un  dispositif 
OTH-S  (radar  a  ondes  de  surface)  vient 
principalement  des  contraintes  liees  a  la 

propagation  ionospherique.  La  distance  entre 

l'emetteur  et  la  zone  a  surveiller  doit  etre  comprise 
typiquement  entre  800  et  2500  km.  Par  ailleurs,  les 
fluctuations  temporelles  du  milieu  de  propagation 
imposent  une  reactualisation  frequente  des 
parametres,  notamment  de  la  frequence  porteuse. 

Le  second  inconvenient  tient  a  la 

complexity  du  traitement  de  signal.  Pour  une 
exploitation  optimale  des  capacites  dc  detection  et 
localisation  du  dispositif,  des  procedures  de  filtrage 
adaptatif  angulaire  doivent  etre  mises  en  oeuvre. 
Toutefois,  la  charge  de  calcul  supplementaire, 
comparee  a  ce  que  requiert  le  traitement  classique 
d'un  radar  OTH-S  ,  ne  doit  pas  etre  consideree 
comme  un  obstacle,  eu  egard  aux  performances  sans 
cesse  croissantes  des  processeurs  de  calcul 
disponibles  actuellement  sur  le  march# 


2.  PERFORMANCES 

2.1.  Influence  de  la  propagation  et  de 
1'  environnement 

La  propagation  et  l'environnemcnt  jouent  un  role 
determinant  dans  revaluation  des  performances  du 
radar  Nostramarine. 

Les  conditions  de  propagation  par  onde  de 
del  ,  ainsi  que  la  contrainte  de  recherche  d’une 
frequence  claire  imposent  dans  la  pratique  le  choix 
d’une  largeur  de  bande  de  fonctionnement. 

La  propagation  par  onde  de  mer,  le  long  du 
trajet  cible-recepteur  ne  necessite  aucune  prediction 
particuliere.  Le  fonctionnement  en  polarisation 
verticale  est  en  revanche  necessaire.  Les  pertes  de 
propagation  presentent  un  comportement  favorisant 
l’utilisation  de  basses  frequences.  Au  dela  de  15 
MHz,  et  pour  des  distances  importantes  (>  300  km) 
les  pertes  deviennent  tres  importantes  (>  30  a  50 
dB).  L'etat  de  la  mcr,  par  la  rugosite  de  surface  qu'il 
induit,  tend  a  augmenter  encore  les  pertes  de 
propagation,  et  ce,  d'autant  plus  que  la  frequence  de 
fonctionnement  est  elevee  (notamment  au  dela  de 
10  MHz). 

2.2.  Modelisation 

Les  modeles  de  propagation  utilises  pour 
1'evaluation  des  performances  theoriques  sont  d'une 
part  le  modele  de  Bradley-Dudeney  pour  la 
propagation  ionospherique  [1],  d'autre  part  le 
modele  de  Bremmer  pour  la  propagation  par  onde 
de  mer[2,3,4]  .  Le  modele  Bradley-Dudeney 
consiste  en  un  trace  de  rayon  a  partir  d'un  modele 
de  profil  d'indice  de  refraction  a  2  couches 
paraboliques.  Le  modele  de  Bremmer  procede  au 
calcul  de  la  serie  des  residus  du  potentiel  de  Hertz, 
au  dessus  d'un  interface  spherique  lisse.  L'etat  de  la 
mer  et  la  rugosite  des  vagues  sont  prises  en  compte 
sous  la  forme  d'une  impedance  de  surface  modifiee, 
d'apres  les  travaux  de  Barrick[5],  A  titre  d'exemple, 
la  prise  en  compte  d'un  etat  de  mer  5  sur  une 
propagation  par  ondes  de  surface  operant  a  1 5  MHz 
conduit  a  des  pertes  supplementaires  de  6  dB  a  70 
km. 

Le  bruit  radioelectrique,  quantifie  par  la 
notion  de  facteur  de  bruit,  est  delicat  a  modeliser. 
Le  bruit  fluctue  dans  le  temps,  et  ses  proprietes 
dependent  de  l’emplacement  geographique;  les 
zones  rurales  sont  a  privilegier  devant  les  zones 
urbaines  qui  sont  la  source  de  bruits  industriels 
contribuant  a  une  augmentation  significative  du 
facteur  de  bruit  ambiant.  Des  differences  de  facteur 
de  bruit  de  10  a  20  dB  entre  un  site  ‘calme’  et  un 
site  ‘bruyant’  sont  courantes  en  gamme  HF.  En 
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environnement  calme,  le  facteur  de  bruit  moyen 
peut  etre  modelise  simplement  par  l’expression 
suivante:  F(dB)  =  56  -  25  logio(fMHz)  ■  Cette 
expression  valable  en  dessous  de  30  MHz  pour  une 
antenne  courte  non  directive,  est  coherente  avec  les 
informations  connues  du  CCIR  . 


2.3.  Bilan  energetique 

En  l'absence  d'interferences,  le  bilan 
energetique  pour  le  radar  Nostramarine  s'ecrit: 


P,  G  (l  o  « 

<4 


En  preambule  a  un  examen  detaille  des 
performances,  il  est  interessant  de  presenter  les 
ordres  de  grandeur  des  parametres: 


f=c/l 

Frequence 

12  MHz 

F 

Facteur  de  bruit  exterieur 

30  dB 

Pe-Ge 

PIRE  OTH 

10  MW 

a 

Facteur  de  forme 

1 

Diono 

Distance  ionospherique 

1500  km 

Dos 

Distance  cible  -  reception 

100  km 

Ljono 

Pertes  onde  de  ciel 

13  dB 

L-os 

Pertes  onde  de  sol 

15  dB 

Gr 

Gain  reception 

(Nostramarine) 

directivite 

15  dB 
20° 

cr 

SER  de  la  cible 

10  m2 

Tint 

Duree  d'integration 

4  sec. 

Le  rapport  signal  a  bruit  est  alors  egal  a  : 

S/B  [Nostramarine]  =  18dB 
II  est  interessant  de  comparer  ce  resultat  au  bilan 
energetique  que  l’on  obtiendrait  a  l'aide  du  radar 
OTH-B  utilise  de  maniere  conventionnelle,  ou  a 
l’aide  d'un  radar  OTH-S  monostatique.  Les  donnees 
complementaires  suivantes  sont  necessaires  (ces 
donnees  sont  toujours  a  considerer  comme  des 
ordres  de  grandeur): 


p;< 

PIRE  -  OTH-S 

100  kW 

a 

Facteur  de  forme 

'/4 

Gr 

Gain  reception  -  OTH-B 
directivite 

28  dB 

3  ° 

S_ 

B 


1  OTH-S 


Pe  Ge-Gr  °T,na 
(4*?DlL2os  fktq 


=  14  dB 


P,  G,  Gr  ° 


=  AdB 


M)th-b  (4 k)7,  n  „  •  L  '  FKT{) 

v  7  *Stono  ^lono  u 


Sur  le  plan  energetique,  Nostramarine  est 
equivalent  a  un  radar  monostatique  dont  la 
puissance  moyenne  rayonnec  serait  de  l’ordre  de  la 
centaine  de  kW.  Outre  l'economie  d'un  systeme 
d'emission  encombrant  (antennes  accordees 
necessitant  une  infrastructure  lourde),  le  dispositif 
Nostramarine  peut  etre  con<ju  a  partir  d’un  systeme 
d'antennes  de  reception  legeres,  facilement 
transportables  et  deplagables.  Par  ailleurs  les 
performances  obtenues  avec  le  radar  OTH-B  sont 
tres  en  retrait  des  performances  obtenues  avec 
Nostramarine.  En  effet,  la  difference  de  distance,  le 
long  du  trajet  retour  [cible-recepteur],  soit  1500  km/ 
100  km  n’est  que  partiellement  compensee  par  la 
difference  de  gain  en  reception  (28  dB/15  dB). 
Dans  le  cas  considere,  les  pertes  de  propagation 
sont  egalement  plus  elevees  sur  le  parcours  de 
l’onde  de  ciel  que  sur  le  parcours  de  l’onde  de  sol. 

Pour  ce  qui  concerne  la  localisation  des 
cibles,  les  resolutions  en  distance  sont  comparables, 
qu’il  s’gisse  des  radars  OTH-S,  OTH-B  ou  OTH- 
S/B  (Nostramarine).  La  resolution  transversale  est 
liee  a  la  resolution  azimutale  du  radar  et  a  la 
distance  des  cibles  : 

-  de  l’ordre  de  20°  a  100  km  pour  les  OTH-S  et 
Nostramarine,  soit  35  km, 

-  de  l’ordre  de  3°  a  1100  km  (distance  au  sol)  pour 
les  OTH-B,  soit  50  km. 


2.4.  Modelisation  de  la  couverture 

Un  logiciel  de  calcul  de  la  couverture  radar 
selon  le  precede  Nostramarine  a  ete  mis  au  point  a 
l'Onera.  Les  figures  4  et  5  fournissent  quelques 
exemples  de  couvertures  simulees.  Les  couvertures 
sont  calculees  a  partir  d’une  modelisation  du 
rapport  signal  a  bruit  en  environnement  calme;  la 
vitesse  des  cibles  considerees  est  telle  que  les  echos 
de  fouillis  n’en  perturbent  pas  la  detection.  L’effet 
de  secteur  aveugle,  qui  peut  apparaitre  du  fait  de 
[’elimination  angulaire  du  trajet  direct  n’a  pas,  pour 
1’ instant  ete  modelise. 

La  figure  4  est  relative  a  l'utilisation  d'un  emetteur 
OTH  de  type  Nostradamus  implante  a  l'ouest  de 
Paris.  Le  dispositif  de  reception  est  implante  sur  la 
cote  mediterraneenne,  a  proximite  de  Toulon.  Les 
caracteristiques  de  la  simulation  sont  resumees  dans 
le  tableau  ci  dessous.  La  portee  theorique  sur  cible 
aerienne  est  de  l'ordre  de  170  km. 

La  figure  5  est  relative  a  l'utilisation  d'un  emetteur 
HF  non  cooperatif.  Ici,  l'emetteur  de  Simferopol 
permet  d'obtenir  des  portees  de  l'ordre  de  115  km 
sur  cible  aerienne. 
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fig.  4  :  exemple  de  couverture  radar  obtenue  a  l’ aide  d’un  emetteur  OTH 

-  (a) :  densite  de  puissance  incidente  ;  (b) :  couverture  radar  (portee  de  I’ordre  de  170  km) 
-frequence  :  14.00  MHz,  puissance  10  MW, 

-  reseau  de  reception  a  12  antennes  ( secteur  80°),  integration  coherente  sur  10  secondes 

-  cible  de  SER  2  m2,  altitude  30  m,  mer  calme 

-  caracteristiques  de  l’ ionosphere  :  hauteur  de  la  couche  E :  100  km  (  Fc=3  MHz) 

hauteur  de  la  couche  F :  320  km  (  F(-10  MHz ) 


fig- 5 


exemple  de  couverture  radar  obtenue  avec  l’ emetteur  de  radiodiffusion  de  Simferopol 


-  (a)  :  densite  de  puissance  incidente  ;  (b) :  couverture  radar  ( portee  de  I’ordre  de  115  km) 
-frequence  :  15.38  MHz,  puissance  1MW,  (heure  TU :  I5H00) 

-  reseau  de  reception  a  12  antennes  (secteur  80°),  integration  coherente  sur  10  secondes 

-  cible  de  SER  5  m2,  altitude  15  m,  mer  calme 

-  caracteristiques  de  l  ’ionosphere  :  hauteur  de  la  couche  E :  100  km  (  Fc=3  MHz) 

hauteur  de  la  couche  F :  320  km  (  Fc-10  MHz) 
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fig.  6  :  Exemple  de  repartition  d}  emetteurs  non  cooperatifs  (PIRE  >  100  Kw  -  octobre  96) 

La  figure  6  montre  une  carte  des  emetteurs 
de  radiodiffusion  puissants  (  PIRE>  100  kW)  et 
disponibles  a  une  periode  donnee.  Le 
fonctionnement  en  reception  sur  plusieurs 
frequences  (frequences  d’emetteur  OTH  cooperatifs 
et  frequences  d'emetteur  de  radiodiffusion)  confere 
au  dispositif  Nostramarine  des  proprietes 
multistatiques.  La  detection  des  cibles  a  partir  de 
plusieurs  emetteurs  va  permettre  de  confirmer  les 
plots  de  detection  et  affiner  la  localisation  des 
cibles. 

3  .  TRAITEMENT  DU  SIGNAL 
3.1.  Traitement  de  detection 

Le  traitement  des  signaux  re?us  consiste  a 
separer  les  contributions  du  trajet  direct  (ondes  de 
ciel)  et  des  trajets  diffus  presumes  contenir  les 
echos  de  cibles  mobiles. 

Du  fait  de  la  configuration  bistatique,  la 
forme  d'onde  radar  peut  etre  continue;  la  bande  du 
signal  emis  est  en  revanche  limitee  a  la  largeur  de 
bande  disponible  autour  de  la  frequence  de 
fonctionnement  du  radar.  Typiquement,  des  largeurs 
de  bande  comprises  entre  10  et  30  kHz  peuvent 
etre  obtenues,  et  ce,  d'autant  plus  facilement  que  la 
frequence  porteuse  se  situe  dans  le  haut  de  la  bande 
decametrique  (  >  15  MHz).Le  traitement  en 
reception  comprend,  apres  le  codage  et  le  filtrage  de 
Hilbert  des  signaux,  le  traitement  d'antenne,  la 


compression  d’impulsion  et  l'analyse  Doppler.  La 
figure  7  presente  le  synoptique  general  du 
traitement . 


fig.  7  :  synoptique  general  du  traitement  de  signal 
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Le  traitement  d'antenne  est  necessaire  pour  detecter 
et  localiser  les  cibles;  son  role  est  de  diminuer  la 
puissance  du  trajet  direct  incident  afin  de  restituer 
au  voisinage  de  la  cible  un  contraste  suffisant. 

Le  traitement  angulaire  est  de  type  adaptatif,  la  mise 
en  place  d'un  "zero"  dans  la  direction  du  trajet 
direct  permettant  d'ameliorer  de  30  a  40  decibels  le 
taux  de  contraste  de  la  cible.  Le  bruit  de  phase,  qui 
accompagne  inevitablement  les  signaux  emis  est 
egalement  elimine  par  le  traitement  angulaire.  Si 
Ton  note  Sn(t)  le  signal  re$u  sur  le  capteur  n,  alors 
le  traitement  adaptatif  angulaire  consiste  a 
determiner  le  vecteur  des  ponderations  W  =  {wn}  de 
fa?on  a  maximiser  en  sortie  le  rapport  signal  a 
[bruit  +  interferences].  Ce  probleme  est  un 
probleme  classique  de  minimisation  sous  contrainte 
que  Ton  peut  exprimer  sous  la  forme: 

min(WM  RW)  avec  R  =  (SbSb) 

W'D  =  1 


ou  D  est  le  vecteur  directionnel  de  la  cible  et  R  la 
matrice  de  covariance  des  interferences  (notees  Sb). 
La  solution  a  ce  probleme  de  minimisation  est : 


w  = 


R~'D* 

D*R''D* 


Le  vecteur  D  etant  specifique  d’une  direction 
particuliere,  le  balayage  angulaire  en  mode  veille 
impose  de  calculer  pour  chaque  valeur  angulaire  0: 
R~’D*(0) 


W(0)  =  - 


D'(0)R  D*(0) 


La  matrice  de  covariance  R  est  estimee  a  partir  des 
donnees  elles  memes,  car  les  signaux  de  bruit,  notes 
Sb  ne  sont  pas  disponibles  separement.  La  matrice 
est  calculee  en  moyennant  sur  un  nombre  de 
recurrences  radar  Nr  (  Nr>2N  dans  la  pratique)  les 
produits  de  correlation  de  signaux  preleves  de 
preference  dans  une  case  distance  differente  de  la 
case  distance  presumee  de  la  cible.  Cette  precaution 
permet  d’eviter  de  degrader  le  niveau  de  la  cible 
lorsqu’elle  rentre  dans  l’estimation  de  la  matrice  de 
covariance.  En  revanche,  si  la  cible  se  trouve  dans 
la  direction  du  trajet  direct,  elle  se  trouve  eliminee. 
L  ’elimination  angulaire  du  trajet  direct 
ionospherique  peut  ainsi  conduire  a  un  secteur 
aveugle  en  detection.  Ce  phenomene  peut  etre 
contourne  en  utilisant  un  reseau  surfacique 
(presentant  de  la  directivite  en  site)  ou,  plus 
simplement,  un  reseau  a  2  rideaux  d’antennes 
correctement  orientes.  Dans  le  cas  represente  sur 
la  figure  8  (cas  b),  Lelimination  du  trajet  direct  a 
site  et  gisement  non  nuls  ne  conduit  a  aucun  secteur 
aveugle  a  site  nul. 


direction  aveugle  a  site  nul 


(b) 


Trajet  direct 


pas  de  direction  aveugle 
a  site  nul 


•  • 


•  • 

•  • 


fig.  8  :  choix  d’une  configuration  permettant  de 
limiter  l  'ejfet  de  secteur  aveugle 

La  compression _ d’impulsion, 

conventionnelle,  repose  sur  la  correlation  des 
signaux  requs  avec  une  replique  du  code  6mis.  La 
replique  utilisee  peut  egalement  provenir  du  trajet 
direct  lui  meme,  lorsque  ce  dernier  presente  un  bon 
rapport  signal  a  bruit.  Cette  technique  permet  de 
compenser  une  grande  partie  des  derives 
ionospheriques  (retard  de  groupe  et  doppler  ),  dont 
on  peut  faire  l’hypothese  qu'elles  sont  semblables  le 
long  du  trajet  direct  et  des  trajets  emetteur-cibles 
dans  la  zone  locale  consideree.  Cette  compensation 
automatique  evite  la  correction  a  posteriori  des  biais 
ionospheriques. 

La  compression  d'impulsion  est  suivie 
d'une  analyse  Doppler  par  FFT,  technique  classique 
et  peu  couteuse  en  temps  de  calcul.  Les  signaux, 
avant  analyse  Doppler,  sont  ponderes  (fenetre  de 
Hanning  ou  de  Blackman).  L'analyse  Doppler 
permet  de  separer  les  dchos  de  cibles  mobiles  du 
trajet  direct  residuel  et  du  fouillis.  Ce  dernier,  apres 
analyse  Doppler  fine  (integration  coherente  sur 
plusieurs  secondes)  presente  autour  de  zero  un 
etalement  de  l’ordre  du  Hertz.  Le  traitement 
Doppler  ameliore  le  contraste  entre  les  cibles 
mobiles  et  le  fouillis  +  trajet  direct,  d'environ  60  a 
70  dB. 

L'amelioration  globale  du  contraste  entre 
les  echos  de  cibles  et  les  signaux  a  Doppler  nul  est 
ainsi  de  l'ordre  de  100  dB,  ce  qui  compense 
parfaitement  la  dynamique  initialement  elevee  entre 
les  signaux  utiles  et  les  signaux  parasites. 


3.2.  Association  de  plusieurs  detections 

L'utilisation  simultanee  de  plusieurs  emetteurs  est 
envisageable.  Les  emetteurs  de  radiodiffusion 
puissants  sont  utilisables  grace  a  leur  porteuse 
stable,  propice  a  une  analyse  Doppler  coherente.  En 
revanche,  l'absence  de  'code  radar'  (  ou  forme 
d'onde  recurrente)  compromet  la  localisation  en 
distance  des  cibles.  Les  plots  de  mesure  sont  done 
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constitues  de  I'azimut  et  du  Doppler  de  la  cible 
relatif  a  chacun  des  emetteurs  envisages. 
L'association  des  plots  (et  plus  precisement  des 
Doppler)  en  vue  de  la  localisation  plane  des  cibles  a 
deja  ete  utilisee  dans  le  cas  de  la  detection  a  l’aide 
des  emetteurs  de  television  analogique  [6]  .  Le 
principe  repose  sur  la  resolution  d’un  systeme 
d'equations  non  lineaires  reliant  les  frequences 
Doppler  de  la  cible  ses  coordonnees  planes  et  son 
vecteur  vitesse: 


avec  : 

(x.y)  :  coordonnees  de  la  cible  C 

(Vx,Vv)  :  vecteur  vitesse  de  la  cible 
(xen,ye„)  :  coordonnees  de  l’emetteur  n 
(xr,y,)  :  coordonnees  du  site  de  reception  R 

d(M|,IVL)  :  distance  de  groupe  entre  M,  et  M2 

Dans  le  cas  d'une  localisation  plane  (x,y), 
l’utilisation  de  4  emetteurs  est  necessaire.  Pour  que 
la  localisation  soit  correcte,  il  est  indispensable  de 
separer  au  prealable  les  cibles  en  vue  d'associer  les 
Doppler  correspondant  a  la  metric  cible.  Ceci  est 
envisageable  grace  aux  capacites  de  separation 
angulaire  du  reseau  de  reception 


4.  RESULT  ATS  EXPERIMENTAUX 

Des  campagnes  de  mesures  ont  ete  menees 
dans  le  but  de  demontrer  la  faisabilite  du  procede 
Nostramarine.  Des  essais  ont  d'abord  ete  conduits  a 
1’Onera-Palaiseau  a  l'aide  de  3  antennes,  en 
presence  d'avion  civils  (Orly)  en  utilisant  un 
emetteur  de  radiodiffusion  situe  a  Madrid;  ces 
essais  ont  permis  de  valider  les  differentes  etapes  du 
traitement  de  signal.  Des  campagnes  de  mesure 
specifiques  ont  eu  lieu  sur  le  site  de  Losquet 
(contrat  DRET),  en  cooperation  avec  le  CNET,  et 
au  CEM  (  contrat  SPAe).  Au  cours  de  ces  deux 
campagnes  de  mesures,  les  capacites  de  detection 
au  dela  de  l’horizon  du  dispositif  Nostramarine  ont 
pu  etre  etudiees,  les  emetteurs  utilises  etant 
Nostradamus  ou  des  emetteurs  de  radiodiffusion 
puissant. 

La  figure  9  presente  un  resultat  de 
detection  (piste  Doppler-temps)  obtenu  depuis  le 
site  de  Losquet  (CNET)  a  partir  de  l'emetteur  de 
radiodiffusion  de  Kiev.  La  frequence  porteuse  est 
de  l’ordre  de  15  Mhz;  la  cible  evolue  a  une  altitude 
de  300  pieds  a  une  distance  de  75  km.  Le  reseau  de 
reception  comprend  8  antennes  isotropes  en 
gisement  a  polarisation  verticale.  L’horizon 
radioelectrique  (distance  maximale  de  visibilite 
d’un  radar  conventionnel)  est  de  l’ordre  de  45  km. 


dic.i;n)  d(C.k) 
y-  V  -Y. 


d(C.I-.„)  d(C.R) 


fig. 9  :  exemple  de  resultat  de  detection  trans horizon  (  emetteur  de  radiodiffusion  de  Kiev) 
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CONCLUSION 

Alternative  au  radar  transhorizon  a  ondes 
de  surface,  le  dispositif  Nostramarine  presente  au 
niveau  de  la  zone  a  surveiller  l’attrait  d’un  systeme 
passif,  peu  vulnerable  et  facilement  deployable.  Le 
fonctionnement  suppose  de  combiner  un  mode  de 
propagation  par  onde  de  ciel  (pour  eclairer  les 
cibles  basse  altitude)  et  un  mode  de  propagation  par 
onde  de  surface.  Le  fonctionnement  en  liaison  avec 
un  emetteur  radar  OTH  ou  un  emetteur  de 
radiodiffusion  est  done  necessaire.  L’opportunite 
fournie  par  la  multiplicity  des  emetteurs  de 
radiodiffusion  puissants  (puissance  pouvant 
atteindre  le  megawatt)  a  ete  introduite  d’un  point  de 
vue  multistatique.  Le  systeme  de  reception  et  de 
traitement  peut  etre  agence  de  maniere  a  traiter 
simultanement  les  signaux  provenant  de  plusieurs 
emetteurs.  Les  emetteurs  radar  (OTH-B),  par  leur 
forme  d’onde  specifique,  permettent  une 
localisation  en  distance;  les  emetteurs  de 
radiodiffusion  ne  permettent  pas  de  localiser  en 
distance  les  cibles.  Par  contre,  un  traitement 
dissociation  des  frequences  doppler,  obtenues  sur 
une  meme  cible  a  partir  de  plusieurs  emetteurs 
d’opportunite  devrait  permettre  d’envisager  une 
localisation  plane  de  cette  demiere. 

Plusieurs  campagnes  de  mesures  ont  deja 
ete  menees  dans  le  but  de  valider  le  fonctionnement 
du  dispositif  Nostramarine.  Un  resultat  dc  detection 
obtenu  a  l’aide  d’un  emetteur  de  radiodiffusion  a 
permis  d’illustrer  les  capacites  de  detection 
transhorizon  du  radar. 

Au  dela  de  ses  performances  de  detection 
transhorizon,  Nostramarine  est  un  radar  passif 
reconfigurable,  pouvant  tirer  partie  du 
multistatisme,  et  dont  l’interet  repose  sur  une  faible 
vulnerability,  un  deployment  rapide  et  discret 
d’antennes  legeres.  Enfin,  le  cout  du  dispositif  est 
egalement  faible,  compare  aux  systemes  radar 
traditionnels:  le  cout  est  conditionne  pour  une 
grande  partie  par  le  calculateur  de  traitement  de 
signal,  ce  dernier  incluant  des  processeurs 
classiques,  disponibles  sur  etagere  avec  un  rapport 
cout/performances  evoluant  favorablement  au  cours 
du  temps. 
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PAPER  No.  36 


DISCUSSOR’S  NAME:  C.  Goutelard 


COMMENT/QUESTION: 

Vous  utilisez  la  conjugaison  de  phase  pour  focaliser  votre  faisceau,  emis  d’un  OTH,  sur  le  site  de 
reception.  C’est,  je  pense,  une  methode  tres  bonne  mais  dans  le  cas  present,  1’ ionosphere,  le 
milieu  de  propagation  est  non-reciproque  et  de  plus  la  propagation  est  multimode.  Avez-vous 
examine  l’influence  de  ces  effets? 

(You  use  phase  conjugation  to  focus  your  beam,  emitted  by  an  OTH,  on  the  receiving  site.  I  think 
this  is  a  very  good  method,  but  in  the  present  case  the  ionosphere,  the  propagation  medium,  is 
non-reciprocal  and  what  is  more,  the  propagation  is  multimode.  Have  you  looked  at  the 
influence  of  these  effects?) 


AUTHOR/PRESENTER’S  REPLY: 

La  comparaison  de  phase  n’a  pour  V  instant  pas  ete  testee  experimentalement  dans  le  cadre  de 
transmissions  HF  ionospheriques.  En  presence  de  multitrajets  de  niveaux  differents,  la  technique 
de  conjugaison  de  phase  conduit  a  selectionner  le  trajet  dont  le  niveau  et  preponderant.  Dans  la 
pratique,  le  faisceau  est,  pour  le  moment,  postionne  de  maniere  manuelle,  et  la  determination 
d’un  trajet  preponderant  ne  semble  pas  poser  de  probleme.  On  peut  ainsi  esperer  que  la 
conjugaison  de  phase,  qu’il  reste  a  implementer,  sera  efficace  dans  la  majorite  des  cas.  Mais  les 
problemes  que  vous  citez,  specifiques  du  milieu  ionospheriques  doivent  etre  pris  en  compte. 

( For  the  moment,  phase  comparison  has  not  been  experimentally  tested  in  the  framework  of  HF 
ionospheric  transmissions.  In  the  presence  of  multipaths  of  different  levels,  the  phase 
conjugation  technique  involves  selection  of  the  path  with  the  predominant  level.  In  practice,  the 
beam  is  for  the  moment  positioned  manually  and  the  determination  of  the  predominant  path  does 
not  seem  to  pose  any  problem.  Hopefully  then,  phase  conjugation,  which  remains  to  be 
implemented,  will  be  effective  in  most  cases.  However,  the  problems  which  you  mention,  which 
are  specific  to  the  ionospheric  medium,  should  be  taken  into  account.) 
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THE  NATO  DATA  FUSION  DEMONSTRATOR  PROJECT 


J  M  Skinner 

Command  Systems  Department 
Defence  Evaluation  and  Research  Agency 
Malvern,  WR14  3PS,  UK 


SUMMARY 

This  paper  defines  data  fusion  in  the  context  of  the  land  battle 
and  the  military  intelligence  cycle,  it  describes  the  NATO  Data 
Fusion  Demonstrator  concept  and  the  key  aspects  of  the  current 
design  and  concludes  with  a  description  and  evaluation  of  the 
results  achieved  to-date. 

1  INTRODUCTION 

1.1  The  Problem 

Land  commanders  at  all  levels  need  timely  and  accurate 
intelligence  on  the  enemy’s  locations,  strength,  activities, 
subordination  and  intentions  in  order  to  manage  their  battle 
space  effectively  and  get  within  the  Opposing  Forces  (OPFOR) 
reaction  time.  The  ever  increasing  amount  of  sensor  data, 
Initial  Processing  Site  (IPS)  information  together  with 
information  and  intelligence  from  subordinate,  peer  and 
superior  commands  as  well  from  other  sources  and  agencies 
now  makes  it  increasingly  difficult  for  manual  All  Source  Cells 
(ASC)  to  process  this  wealth  of  information  sufficiently  quickly 
and  accurately  to  meet  their  commander’s  intelligence  needs. 


1.2  The  Intelligence  Cycle 


The  processes  carried  out  by  an  ASC  are  defined  by  the 
Intelligence  Cycle  shown  in  Figure  1.  The  first  stage,  Direction, 
consists  of  receiving  the  commander’s  intelligence 
requirements,  determining  the  information  needed  to  fulfil 
those  requirements,  preparing  a  collection  plan,  tasking  sources 
and  agencies  and  monitoring  their  activity.  The  second  stage, 
Collection,  consists  of  the  exploitation  of  the  sources  of 
information  by  the  collection  agencies  and  the  delivery  of  the 
collected  information.  The  third  stage.  Processing,  consists  of 
collation,  evaluation  and  interpretation.  The  DFD  is  concerned 
only  with  the  evaluation  phase  which  may  be  sub-divided  into 
the  processes  of  classification,  correlation,  association  and 
aggregation.  Interpretation,  which  includes  predicting  the 


OPFOR’ s  objectives  and  future  courses  of  action  is  outside  the 
scope  of  the  current  project.  Dissemination,  the  fourth  stage  of 
the  cycle,  consists  of  providing  the  commander  and  others  with 
the  intelligence  they  have  requested  and  need  to  fulfil  their 
roles  effectively. 

1.3  The  Definition  of  Data  Fusion 

For  the  purposes  of  this  paper,  data  is  defined  as  the  output  of  a 
transducer  or  sensor  produced  by  a  change  in  its  surveillance 
environment.  Information  is  the  classification  of  that  data  by 
the  application  of  collateral  knowledge  and  inferencing  in  order 
to  recognise  or  identify  the  detected  entity,  activity  or 
occurrence.  Finally,  intelligence  is  produced  by  interpreting 
the  significance  of  that  information  by  the  application  of 
inferencing  together  with  collateral  knowledge  of  the  enemy 
commander’s  Order  of  Battle  (ORBAT),  doctrine  and  potential 
deception  tactics  together  with  deductions  of  how  his  activities 
may  be  constrained  by  the  terrain  and  weather  conditions,. 

As  can  be  seen  from  the  foregoing,  the  application  of  the  term 
data  fusion  to  this  project  could  be  regarded  as  a  little 
misleading  as  the  DFD  is  concerned  primarily  with  the  fusion 
of  information  from  IPS  and  other  sources  rather  than  with  the 
fusion  of  raw  sensor  data. 

Nevertheless  data  fusion  is  the  commonly  accepted  term  for  the 
processes  of  combining  data  and  or  information.  Even  so  there 
are  still  many  different  definitions  of  data  fusion.  However,  in 
the  land  military  context,  the  product  of  data  fusion  should  be 
regarded  as  an  integral  and  vital  component  of  the  intelligence 
cycle  and  hence  battle  management  and,  for  the  purposes  of  the 
DFD  project,  (Ref.  1)  is  defined  as:- 

The  process  of  creating  a  timely,  consistent  and 
effective,  task  orientated  perception  of  the  world 
from  all  source  data  and  information  which  may  be 
incomplete  and/or  inaccurate  and/or  misleading 
and/or  untimely. 

Data  fusion  is  achieved  by  some  or  all  of  a  series  of  processes 
in  which  the  input  data  or  information  is  collated,  classified, 
correlated,  associated  and  aggregated  to  provide  a  reduced  data 
set  whose  information  content  is  greater  than  that  of  its 
constituent  parts.  Thus  the  simultaneous  display  of  data  is  not 
strictly  data  fusion,  nor  is  the  process  of  limiting  or  discarding 
data  in  an  arbitrary  process.  The  essence  of  data  fusion  is  data 
reduction  with  synergy. 

The  DFD  definitions  of  the  processes  which  go  to  make  up  data 
fusion  are  based  on  current  British  and  NATO  usage  and  are  as 
follows. 

Collation.  The  grouping  together  of  related  items  of 
information  to  provide  a  record  of  events  and  facilitate  further 
processing.  Collation  is  the  precursor  to  classification  and  only 
involves  collecting  similar  sets  of  data  and  information  from 
similar  sources. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar",  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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Classification.  The  inferencing  process  which,  using  collateral 
information,  classifies  or  recognises  or  indentifies  the  reported 
entity,  activity  or  occurrence. 

Correlation.  The  process  which  combines  data  and 
information  from  similar  sources  about  the  same  entity,  activity 
or  occurrence. 


and  concludes  with  their  deployment  into  combat  battle 
formation. 

The  initial  phase  of  the  advance  is  conducted  in  radio  silence 
and  the  Blue  Commander’s  information  gathering  capability  is 
further  hindered  as  he  is  not  allowed  to  deploy  any  sensors  or 
reconnaissance  elements  across  the  international  boundary. 


Association.  The  logical  blending  of  related  but  dissimilar  sets 
of  correlated  information,  from  a  variety  of  dissimilar  sources, 
about  the  same  entity,  activity  or  occurrence.  This  is  the 
process  which  some  workers  call  data  fusion.  It  may  also  be 
called  picture  fusion  when  the  process  combines  the  separate 
ELINT,  COMINT,  IMINT  and  HUMINT  pictures. 

Inferencing.  The  use  of  collateral  knowledge  to  deduce  or  infer 
additional  information  about  an  entity  or  an  activity  or 
occurrence.  Inferencing  can  occur  in  all  the  processes  of  data 
fusion  and  in  conjunction  with  correlation  and  association  can 
confirm  or  improve  the  recognition  or  identification  and 
location  of  an  entity  or  class  of  activity  or  occurrence. 

Aggregation.  The  process  which  combines  or  aggregates  those 
associated  entities  that  are  related  to  each  other  in  order  to 
form  parent  entities,  similarly,  the  process  will  aggregate 
parents  to  form  grandparents  and  so  on  to  produce  and  identify 
the  enemy's  hierarchical  structure  or  dynamic  Order  of  Battle 
(ORBAT)  within  the  fusion  system's  areas  of  responsibility  and 
interest.  Thus  the  presence  of  an  HQ  entity  and  three  company 
sized  entities  with  the  appropriate  characteristics  and  within 
sensible  distances  of  each  other  would  lead  the  aggregation 
process  to  deduce  the  existence  of  a  particular  battalion  or  type 
of  battalion.  This  in  turn,  using  the  collateral  Order  of  Battle 
and  doctrine  databases  may  give  rise  to  hypotheses  about  the 
existence  and  role  of  the  parent  regiment. 

1.4  The  Objective  of  the  NATO  DFD  Project 

The  prime  purpose  of  the  DFD  is  to  alleviate  the  intelligence 
analysts’  work  loads  by  demonstrating  the  feasibility  of 
providing  machine  assistance  for  the  creation  of  the  Opposing 
Forces  (OPFOR)  Perceived  Ground  Truth  or  Situation,  that  is 
the  OPFOR  dispositions,  activities  and  subordination,  by  using 
Artificial  Intelligence  (AI),  Knowledge  Based  Systems  (KBS) 
and  other  techniques  to  fuse  the  messages  from  representative 
ELINT,  COMINT,  IMINT  and  HUMINT  IPS  and  other  sources. 

Consequently,  in  the  context  of  the  DFD  project,  data  fusion  is 
confined  to  the  creation  of  the  Perceived  Situation,  that  is  the 
system’s  perception  of  the  ground  truth.  Automatic  or  machine 
assisted  analysis  of  the  perceived  situation  in  order  to  predict 
future  activities  and  objectives  is  outside  the  scope  of  the 
current  project. 

1.5  The  NATO  DFD  Environment 

The  simulation  environment  and  role  chosen  for  the 
demonstrator  is  that  of  a  divisional  ASC,  driven  by  machine 
readable  ADatP-3  messages  (Ref.2)  from  representative 
COMINT,  ELINT,  IMINT  and  HUMINT  Initial  Processing 
Sites.  Each  simulated  IPS  has  a  sensor  model  or  models 
deployed,  to  an  approved  collection  plan,  in  a  simulated  land 
scenario  that  depicts  an  unopposed  divisional  advance 
accompanied  by  guerrilla  deception  activity.  The  scenario 
(Ref.3)  covers  a  24  hour  period  which  commences  with  the 
movement  of  the  Orange  Forces  out  of  their  assembly  area, 
their  advance  along  two  routes,  an  unopposed  river  crossing 


1.6  The  Project  Participants 

The  project  is  governed  by  a  Memorandum  of  Understanding 
(MOU)  (Ref.4)  which  was  signed  by  Canada,  Denmark, 
France,  Germany,  Italy,  The  Netherlands  and  the  UK  in 
September  1992.  However,  Italy  were  unable  to  commence 
work  until  early  1996  and  France  ceased  work  in  April  1996. 
These  two  events  have  necessitated  some  re-structuring  and 
down  sizing  of  the  project.  The  participants  are  currently 
negotiating  an  extension  to  the  MOU  to  enable  the  final  phase 
to  be  completed  by  Italy. 

1.7  The  DFD  Programme 

The  project  was  conceived  as  a  five  year  plan  comprising  of 
three  evolutionary  phases,  Mockup,  Prototype  1  and  Prototype 
2.  The  Mockup  phase,  now  completed,  used  a  preliminary 
scenario  script  and  provided  the  knowledge  and  data  bases,  the 
Human  Computer  Interface  (HCI)  and  the  infra-structure. 

Prototype  1,  due  to  be  evaluated  in  October  this  year,  provides 
the  greater  part  of  the  functionality  of  the  final  system, 
including  all  the  fusion  algorithms,  and  uses  the  full 
International  Scenario  Script  (ISS),  the  Simulated  Ground 
Truth  (SGT)  data  base  and  the  ADatP-3  message  file. 

Prototype  2,  now  due  in  mid  1998,  will  provide  the  final 
system  which  will  include  updates  to  Prototype  1,  an  additional 
classification  algorithm,  based  on  Bayesian  networks,  by 
DDRE,  Denmark,  a  German  aggregation  algorithm  and  the 
additional  sets  of  fusion  algorithms  to  be  provided  by  the 
Italian  participants. 

1.8  The  DFD  System 

To  achieve  the  required  functionality  the  DFD  is  divided  into 
three  systems  or  segments  as  shown  in  Figure  2. 


The  first  segment,  the  Event  Simulation  System  (ESS)  (Ref.5) 
simulates  the  scenario,  the  sensor  models  and  their  deployment 
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and  produces  the  ADatP-3  IPS  message  file  and  the  SGT  data 
base.  The  second  segment,  the  Test  and  Monitor  System 
(TMS),  controls  the  tests  and  passes  the  message  file  to  the 
DFS.  It  was  also  intended  that  the  TMS  would  have  had  the 
facilities  to  compare  the  perceived  situation  produced  by  the 
DFS  with  the  Simulated  Ground  Truth  produced  by  the  ESS. 
However,  the  withdrawal  by  France,  who  was  responsible  for 
the  TMS,  has  resulted  in  these  evaluation  facilities  being 
abandoned  or  at  least,  delayed  until  Prototype  2,  consequently, 
the  TMS  will  not  be  described  any  further  in  this  paper.  The 
third  segment,  the  Data  Fusion  System  (DFS)  (Ref.6)  processes 
and  fuses  the  information  contained  in  the  message  file  and 
produces  and  displays  the  resultant  Perceived  Situation.  In 
addition  to  these  segments  there  is  a  fourth  system,  the 
Communications  and  Supervision  System  (CSS)  which 
manages  the  TMS  and  DFS. 

1.9  The  DFD  Development  and  Target  Environment 

The  major  components  of  the  agreed  development  and  target 
environment  for  the  project  are: 

(a)  Sun  SPARCstation  10/41 

(b)  Solaris  2.5  operating  system 

(c)  C++ 

(d)  Sybase  V4.9.2  and  Objectstore  4.00c  databases 

(e)  TerraLogic  GIS 

(f)  IlogViews  HCI  support 

(g)  IlogRules 

2  THE  EVENT  SIMULATION  SYSTEM  (ESS) 

2.1  The  ESS  Architecture 

The  roles  of  the  ESS  are  twofold,  firstly,  to  provide  the 
messages  necessary  to  exercise  the  DFS  and  secondly,  to 
provide  the  Scenario  Ground  Truth  database,  the  reference 
against  which  the  DFS’s  output,  the  Perceived  Situation 
database  is  assessed.  Figure  3  shows  the  three  major 
components  and  processes  of  the  ESS;  the  Scenario  Handler, 
which  creates  the  detailed  scenario  and  populates  the  SGT 
database,  the  Sensor  Handler  which  deploys  the  sensor  models 
in  the  simulated  scenario  and  creates  sensor  target  data  and  the 
Message  Processor  which  converts  the  target  data  into 
information  formatted  in  ADatP-3  IPS  messages. 


The  inputs  to  the  ESS  consist  of  a  high  level  textual  scenario 
script,  the  International  Scenario  Script  produced  by  Dassault 


Electronique  from  an  earlier  Siemens  AG  scenario,  the 
ORBAT,  a  modified  version  of  the  NATO  Generic  Enemy 
Forces  Catalogue,  the  ADatP-3  message  format  database,  the 
Sensor  Collection  plan  and  the  digital  map  and  route  database 
for  the  scenario  area. 

The  key  outputs  of  the  ESS  are  the  SGT  database  and  the 
Message  File.  The  SGT  database  provides  at  minute  intervals 
the  position,  identity,  subordination,  structure,  activity,  combat 
effectiveness,  boundaries  and  radio  and  radar  emissions  of  each 
independent  unit,  which  are  typically  at  either  platoon, 
company  or  battalion  level. 

The  message  file  consists  of  a  total  of  1053  ENEMY  SITREP, 
RECCEXREP  and  METREP  ADatP-3  messages  containing 
2028  reports  about  Orange  units,  activities  and  occurrences 
over  the  24  hour  scenario  duration.  The  average  report  rate  is 
over  80  per  hour  rising  to  a  peak  of  over  150  per  hour. 

The  subsidiary  ESS  deliverables  are  the  Collection  Plan,  the 
digital  route  database  and  the  scenario  Trafficability  Overlay. 
The  latter  defines  the  scenario  terrain  Go,  Slow  Go  and  No  Go 
areas  and  potential  Avenues  of  Approach. 

2.2  The  ESS  Components 

Unlike  the  remainder  of  the  DFD,  the  ESS  uses  the  DERA 
Carapace  Human  Computer  Interface  (HCI).  Carapace  (Ref.7) 
was  specifically  designed  and  developed  for  intelligence 
applications  and  its  use  in  the  ESS  has  proved  to  be  very  cost 
effective.  The  main  Carapace  components  are  the  Graphics 
Information  Viewer  (GIV),  the  Scenario  Editor  (SE),  the 
Message  Editor  (ME)  and  the  Route  Planner  (RP).  The  Forces 
Editor  (FE)  is  a  PC  based  ORBAT  editor.  The  GIV,  SE,  RP 
and  FE  were  produced  for  DERA  by  Hi-Q  Systems  Ltd.  The 
ME  was  the  Canadian  contribution  to  the  ESS  and  was 
produced  for  the  Defence  Research  Establishment  Valcartier 
(DREV)  by  Le  Groupe  CGI. 

The  core  component  of  the  ESS  is  the  GIV  which  provides 
central  control  of  the  required  area  and  time  frame  and  enables 
the  SE,  ME  and  RP  client  applications  to  have  their  data  and 
information  displayed  either  separately  or  simultaneously  upon 
a  map  background.  Each  client  has  the  ability  to  selectively 
filter  its  output,  making  it  possible,  for  instance,  to  view 
ELINT  messages  produced  by  the  ME  superimposed  upon  the 
SGT  ELINT  emitters  called  up  by  the  SE. 

2.3  ESS  Scenario  Generation 

Figure  4  shows  a  high  level  diagram  of  the  ESS,  the  shaded 
components  are  those  required  to  enter  and  decompose  the 
International  Scenario  Script  and  populate  the  Simulated 
Ground  Truth  database. 

The  scenario  is  entered,  decomposed  and  the  SGT  created  in 
five  stages.  Firstly,  the  locations  and  activities  of  each 
independent  unit  are  manually  transcribed  into  the  Scenario 
Description  text  File  (SDF).  A  unit’s  location  is  updated  at 
each  significant  way  point  on  its  route,  typically  at  30-40 
minute  intervals.  The  SDF  is  then  validated  by  the  parser  and 
entered  into  the  Scenario  Activity  Tables  of  the  Scenario 
Activity  database. 

Secondly,  the  scenario  ORBAT  is  created  by  using  the  Forces 
Editor  (FE)  to  edit  the  NATO  GEFC  to  describe  the 
composition  and  subordination  of  the  scenario  Orange  forces. 
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The  Orange  ORBAT  thus  created  is  then  entered  into  the 
Forces  Tables  of  the  Scenario  Activity  database. 


Thirdly,  the  Route  Planner  is  used  to  define  each  unit’s  route 
between  its  way  points;  a  route  may  be  either  along  the  road 
network  or  cross  country  or  a  mixture  of  the  two. 

Fourthly,  the  SE  uses  this  data  to  decompose  each  unit  down  to 
its  constituent  vehicles  in  order  to  determine  unit  length,  it  then 
moves  each  unit  from  its  assembly  area  onto  its  route  in  the 
correct  sequence  and  updates  the  unit’s  position  every  minute 
as  it  moves  between  its  way  points.  The  SE  warns  the  operator 
of  conflicts  such  as  the  head  of  one  unit  overlapping  the  tail  of 
another  and  provides  an  editing  facility  to  enable  the  operator 
to  resolve  these  and  other  conflicts. 

Finally,  when  all  conflicts  have  been  resolved,  the  SGT 
database  is  populated  with  each  unit’s  identity,  subordination, 
location  and  activity  and  updated  every  minute.  The  SE  also 
provides  facilities  to  view  the  SGT  against  the  map  background 
and  filter  the  data  by  area,  time  period,  unit  type  and  unit  size. 
In  addition  the  SE  allows  the  unit  history  to  be  displayed  as  a 
track  for  any  selected  time  period. 

2.4  ESS  Sensor  Models  and  their  Deployment 

The  ESS  simulates  six  different  types  of  sensor,  a  stand  off, 
airborne  Moving  Target  Indicator  (MTI)  radar,  a  ground  based 
ELINT  Electronic  Support  Measures  (ESM)  system,  a 
COMINT  ESM  system,  a  number  of  Remotely  Piloted  Vehicles 
(RPV)  with  Thermal  Imaging  systems,  a  deployment  of 
Forward  Observers  (FO)  and  an  Air  Reconnaissance  system. 

The  models  are  deployed  to  the  agreed  collection  plan  which 
ensures  that  each  unit  and  significant  event  is  detected  by  at 
least  one  sensor  system.  This  ensures  that  the  DFS  can  be 
evaluated  not  only  with  message  file  that  reports  all  units  and 
activities  but  also  with  depleted  message  files  in  which 
messages  or  sensor  missions  have  been  removed. 

Each  sensor  model  is  a  distinct  Unix  process  that  reads  the 
SGT  database  and  simulates  the  detection  of  scenario  events 
according  to  its  configured  capabilities  and  detection  and 
recognition  performance.  If  necessary  the  models  will 
decompose  the  scenario  activity  within  the  vicinity  of  their 


detection  footprints  into  a  second-by-second  sequence  in  order 
to  produce  the  required  outputs. 

The  ELINT,  COMINT,  Forward  Observer  and  Air 
Reconnaissance  models  directly  generate  their  output  in  the 
form  of  ENEMY  SITREP  ADatP-3  messages  whilst  the  radar 
and  the  RPV  models  produce  binary  output  files  which  are 
converted  into  ADatP-3  messages  by  an  operator  using  the 
Target  Information  Source  Controller  (TISC),  the  GIV  and  the 
ME. 

2.5  ESS  Message  Generation 

The  ESS  components  used  for  message  generation  are  shown 
shaded  in  Figure  5.  The  Message  Format  database  contains  the 
RECCEXREP  format  used  for  the  Air  Reconnaissance 
messages,  the  ENEMY  SITREP  format  used  for  the  RADAR, 
RPV,  ELINT,  COMINT  and  FO  messages  and  the  METREP 
format  used  for  hand  crafted  weather  messages. 


The  function  of  the  ESS  message  processing  capability  is 
firstly,  to  enable  military  analysts  to  view  and  replay  the  MTI 
RADAR  and  RPV  data  on  the  GIV  via  the  TISC.  And  to  create 
information  reports,  in  the  ADatP-3  ENEMY  SITREP  format, 
by  selectively  transferring  data  from  the  GIV  to  the  ME  and 
then  editing  it  to  add  their  classification  of  the  detected  targets 
and  activities. 

Secondly,  to  accept  and  validate  the  ADatP-3  messages  created 
automatically  by  the  other  sensor  models  and  add  them  to  the 
Message  database.  Thirdly,  to  enable  analysts  to  view  the  SGT 
and  create  specialised  reports  such  as  Intelligence  Summaries 
from  Corps  and  flanking  divisions  by  transferring  data  to  the 
ME  and  editing  it.  Fourthly,  to  enable  hand  crafted  messages, 
such  as  the  METREP  weather  reports  to  be  validated  and 
inserted  into  the  Message  database. 

The  messages  from  each  sensor,  including  each  FO,  are  stored 
in  separate  clusters  in  the  Message  database  so  that  different 
Message  Files  can  be  created  which  include  either  all  of  the 
sensor  missions  or  selected  sub-sets.  A  typical  FO  message  in 
the  ENEMY  SITREP  format  reporting  the  detection  of  two 
BTR  armoured  personnel  carries  (APC)  heading  north  east  at 
19.40  Zulu  on  the  4th  of  July  is  shown  below. 
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MSGID  /  ENEMY  SITREP  /  FO  /  65// 

RPTTYPE  /  ENSIT  // 

EFDT  /  041940Z  /  JUL  // 

EGROUP  /  U0067  /ORC  // 

LOCATION/REAL/AREA  /-/-/-/  POINT  /  32UPC8044 178887// 
SOURCE  /  C3  /  OBSN  // 

TIME  /  AT  /  04 1 908ZJUL  // 

ACTIVITY  /  REAL  /  MOVING  // 

DIRCTN/NE// 

EMAT  /  REAL  /  APC  /  BTR/  2  //. 

In  addition  to  the  message  creation  role  the  ME  also  provides 
the  facility  to  selectively  filter  messages  by  source,  area  and 
time  period  and  replay  and  display  the  individual  reports  in 
each  message  graphically  upon  the  map  background.  As  these 
reports  can  be  superimposed  upon  the  SGT  this  facility 
provides  not  only  a  valuable  means  of  validating  the  Message 
File  but  also  a  very  useful  Intelligence  tool  in  its  own  right. 

2.6  ESS  Message  File  Validation 

In  October  1996,  at  The  Netherlands  School  of  Military 
Intelligence,  Ede,  the  DFD  Advisory  Group  held  a  Manual 
Fusion  Exercise  (MFE)  to  validate  the  Prototype  1  ESS 
Message  File.  The  exercise  consisted  of  three  All  Source  Cells, 
one  manned  by  French  Canadian  and  French  military  analysts, 
the  second  by  Danish  and  UK  military  analysts  and  the  third  by 
Dutch  and  German  military  analysts. 

All  three  cells  were  fed  the  ESS  messages  in  the  AdatP-3 
formats  via  a  printer  in  real  time.  The  exercise  was  split  into 
three  eight  hour  sessions  spread  over  three  days,  each  cell  was 
required  to  give  the  Exercise  Control  team  a  commander’s 
briefing  at  the  end  of  each  period. 

All  three  cells  were  successful  in  determining  the  number  of 
enemy  battalions  facing  them  but  were,  in  some  cases, 
reluctant  to  believe  their  conclusions  and  consequently,  failed 
to  assess  accurately  the  level  of  enemy  forces.  This  was  not  due 
to  a  lack  of  information  in  the  Message  File  but  rather  due  to 
the  analysts  developing  a  range  of  hypotheses,  sometimes  based 
on  little  or  no  concrete  evidence. 

The  Advisory  Group  concluded  that  the  exercise  successfully 
validated  the  Message  File  and  that  only  some  minor  changes 
and  additions  were  required.  The  players  also  agreed  that  the 
exercise  provided  them  with  a  credible  scenario. 

The  Prototype  2  (P2)  Message  File  has  now  been  completed, 
delivered  and  validated  by  the  UK  Advisory  Group 
representative.  This  is  the  same  as  the  PI  version  but  with  the 
addition  of  COMINT  and  RECCEXREP  messages  and  a 
limited  number  of  hand  crafted  INTSUMS  from  the  adjoining 
divisions. 

3  THE  DATA  FUSION  SYSTEM 

3.1  The  DFS  High  Level  Architecture 

The  DFS  high  level  architecture  is  shown  in  Figure  6,  the  main 
components  are  the  Asynchronous  Message  Assembly  Unit 
(AMAU)  produced  by  Le  Groupe  CGI,  Quebec.  The 
Correlation,  Aggregation,  Fusion  and  Situation  Determination 
(CAFSD)  system,  the  DFS  Human  Computer  Interface  (HCI) 
and  the  various  data  and  knowledge  bases.  The  DFS  was  or  is 
being  jointly  produced  by  TNO  and  BSO  in  ORIGIN,  in  The 


Netherlands;  Thomson  CSF  in  France;  Siemens  AG,  Domier 
and  Berner  and  Mattner  Systemtechnik  in  Germany;  DREV  and 
Le  Groupe  CGI  in  Canada  and  Sirti,  Italtel  and  Contraves  in 
Italy.  For  historical  reasons  the  designers  of  the  CAFSD  have 
used  slightly  different  terminology  to  that  defined  in  paragraph 
1.3,  here  correlation  refers  to  both  the  correlation  and 
association  processes  and  fusion  to  the  algorithms  used  to 
combine  the  information  about  two  or  more  associated  entities. 
The  purpose  of  the  AMAU  is  to  receive,  process,  validate  and 
decompose  the  constrained  messages  into  their  individual 
reports  and  store  the  results  in  the  Message  Database. 


The  purpose  of  the  CAFSD  is  to  create  the  Perceived  Situation, 
it  does  this  by  converting  each  individual  report  into  a  series 
of  Situation  Elements  (SE)  which  it  stores  in  the  Situation 
Database.  It  then  classifies,  correlates  (associates)  and 
aggregates  the  SEs  using  a  suite  of  classification,  association 
and  aggregation  algorithms  and  stores  and  ranks  all  the 
resultant  hypotheses,  which  represent  all  the  likely  consistent 
situations,  in  the  Situation  Database. 

3.2  The  AMAU 

The  AMAU  receives  the  constrained  messages  from  the  TMS, 
it  parses  each  message,  rejects  messages  which  are  too  stale  or 
refer  to  activities  outside  the  area  of  intelligence  interest  and 
diverts  messages  with  errors  to  the  military  analyst  for 
correction.  It  then  decomposes  each  validated  message  into  its 
individual  reports  in  the  DFS  internal  format  and,  in 
conjunction  with  the  Level  of  Support  (LOS)  component  of  the 
CAFSD,  assigns  a  credibility  rating  to  each  report.  This 
credibility  rating  is  based  upon  the  existing  number  of 
Situation  Elements  which  are  related  to  the  new  report.  The 
AMAU  then  assigns  a  global  reliability  rating  to  the  whole 
message  based  on  the  type  of  source  and  its  reliability  history. 

The  individual  reports  are  then  given  a  priority  rating  based  on 
whether  or  not  they  contain  information  which  meets  the 
commander’s  Priority  Intelligence  Requirements  (PIR).  Finally 
each  processed  report  together  with  its  assigned  number, 
priority  and  reliability  and  credibility  ratings  is  stored  in  the 
Message  Database  and  the  CAFSD  is  informed  of  its  arrival. 

3.3  The  CAFSD 

The  main  components  of  the  CAFSD,  apart  from  the  LOS,  are 
shown  in  Figure  7.  When  informed  of  the  arrival  of  a  new 
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message  and  its  reports,  the  Message  Monster  (Msg  Mon) 
takes  each  report  in  the  message,  converts  it  into  an  Observed 
Situation  Element  (SE),  stores  it  in  the  Situation  Database  and 
informs  the  Correlation  Association  Fusion  (CAF)  system  of  its 
arrival. 


The  CAF  then  retrieves  the  new  Observed  SE  and  integrates  it 
into  the  perceived  situation.  Firstly,  it  uses  the  Basic  and 
ORBAT  data  bases  and  the  Military  Expert  Knowledge  base  to 
classify  the  observed  SE  and  create  a  new  Basic  SE  which  is 
used  in  all  further  processing.  The  classification  algorithm 
uses,  for  instance,  the  information  about  the  number  and  type 
of  vehicles  detected  together  with  the  collateral  information  in 
the  data  and  knowledge  bases  to  derive  the  unit  type, 
specialization  and  command  level.  If  there  is  insufficient 
information  to  do  this  it  creates  an  unidentified  Basic  SE. 
These  processes  are  shown  in  Figure  8. 

Secondly,  it  carries  out  a  pre-association  filtering  to  select  only 
those  SEs  which  might  refer  to  the  same  unit  on  the  battlefield. 
This  is  done  by  taking  into  account  the  age  of  the  information, 
(stale  information  is  not  taken  into  account),  unit  identification 
information,  unit  classification  information  and  the  distance 
between  the  SEs  together  with  their  detection  times.  At  the 
moment  the  location  and  time  checks  do  not  take  account  of  the 
route  network.  By  these  processes  it  determines  whether  or  not 
the  new  SE  can  be  correlated  (associated)  with  any  of  the 
existing  SEs  and  produces  a  Derived  SE  for  each  potential 
correlation  and  computes  the  strength  of  the  correlation  links. 
The  Derived  SE  data  is  produced  by  the  fusion  module  which 
combines  the  information  held  by  each  of  the  correlated  SEs. 

Aggregation  up  to  regiment  level  is  performed  by  an 
aggregation  algorithm  which  uses  disposition  templates  stored 
in  the  Military  Expert  Knowledge  base  to  find  the  best  fit 
between  the  Hypotheses  and  a  template.  The  introduction  of  a 
new  SEs  triggers  the  aggregation  process.  After  aggregation  the 
resulting  Aggregation  Hypotheses  provide  feedback  to  the 
correlation  process  so  that  other  observations  of  the  higher 
level  unit  can  be  identified. 

The  CAFSD  Modification  (MODIF)  module  allows  the  analyst 
to  modify  the  decisions  of  the  fusion  system  in  the  light  of  his 
expert  knowledge  and  experience  by  creating,  modifying  or 


deleting  SEs.  These  changes  are  passed  to  the  CAF  to  enable  it 
to  take  these  changes  into  account. 

The  CAFSD  HCI  and  the  Situation  Object  Selector  (SOS) 
enable  the  analyst  to  query  the  Perceived  Situation  database, 
which  contains  all  the  hypotheses,  and  display  and  filter  the 
most  likely  Consistent  Perceived  Situation  either  for  the  current 
or  any  chosen  historical  time  period.  A  Consistent  Perceived 
Situation  is  one  which  incorporates  all  the  SEs  but  only  those 
hypotheses  which  have  the  highest  probabilities  and  are 
consistent  with  each  other.  Once  a  current  Consistent  Situation 
has  been  selected  the  SOS  maintains  the  display  by  sending 
updates  as  and  when  they  occur.  It  is  also  possible  to  select  and 
display  lower  probability  Consistent  Situations. 

The  Configuration  module  which  will  provide  fine  control  of 
the  CAFSD  has  not  yet  been  integrated. 

3.4  The  Correlation  Algorithm 

The  correlation  algorithm  developed  by  TNO  and  ORIGIN  uses 
the  Like  Situation  Element  First  principle  together  with 
Pairwise  Correlation  and  is  based  on  the  work  of  Miles  (Ref.8). 
In  this  approach  SEs  from  similar  sources  are  correlated  first  as 
their  data  sets  are  similar.  Thereafter  the  new  SE  is  correlated 
in  turn  with  each  of  the  existing  filtered  SEs  and  the  strength  of 
the  correlation  links  determined. 

Level  0  in  Figure  8  shows  the  Observed  SEs,  Level  1  the  Basic 
SEs  and  Level  2  the  Derived  SEs.  In  this  example  the  initial 
correlation  process  has  shown  that  the  A,  B  and  C  Observed 
SEs  refer  to  the  same  entity  and  consequently,  only  one  Basic 
SE  has  been  produced,  whereas  Basic  SEs  have  produced  for 
the  D,  E  and  F  Observed  SEs. 


The  correlation  process  determined  a  weak  correlation  link 
between  (A,B,C)  and  (D)  and  produced  and  maintains  two 
hypotheses  (A,B,C,D)  and  (A,B,C)  and  (D).  There  is  a  strong 
correlation  link  between  (E)  and  (F)  and  again  two  hypotheses 
are  maintained,  (E,F)  and  (E)  and  (F). 

3.5  The  Aggregation  Algorithm 

The  aggregation  engine  developed  by  TNO  and  ORIGIN  uses  a 
library  of  disposition  templates  in  the  Military  Expert 
Knowledge  base  to  match  the  observed  unit  locations  and 
movements  with  known  types  of  disposition  such  as  an 
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unopposed  advance.  The  creation  of  a  new  hypothesis  in  the 
Situation  database  triggers  the  aggregation '  process.  The 
templates  are  positioned  to  the  latest  timed  event  and  all  the 
possible  matches  are  examined  and  the  results  stored  in  the 
Situation  database.  The  selection  of  the  candidate  templates  is 
based  on  the  new  unit  hypothesis,  type  and  command  level  and 
activity.  The  aggregation  hypothesis  strength  is  based  upon  the 
number  of  template  slots  filled  and  the  hypothesis  strengths  of 
the  units  and  their  slots. 

4  THE  DFS  PROTOTYPE  1  PERFORMANCE 

The  Prototype  1  DFD  is  currently  being  integrated  and  tested 
prior  to  being  exercised  in  an  Automated  Fusion  Exercise  in 
October  1997.  Consequently,  the  system  has  not  yet  been 
tested  with  the  full  ISS  and  a  number  of  stability  and 
integration  problems  are  currently  being  resolved.  Initial  tests 
using  the  start  up  situation,  a  series  of  ADatP-3  reports  derived 
from  a  simulated  satellite  mission  over  the  Orange’s  forces 
initial  assembly,  have  shown  that  the  system  can  create  and 
classify  SEs  at  about  50  times  the  speed  of  an  experienced 
analyst  with  a  success  rate  of  about  98%. 

Initial  tests  of  the  correlation  algorithms,  using  the  last  4  hours 
of  the  scenario,  showed  a  capability  of  handling  12  times  the 
peak  ESS  message  rate,  again  with  a  very  high  success  rate. 
More  recent  tests  using  the  P2  ISS  have  shown  that  the  DFS 
can  classify  and  correlate  SEs  for  the  first  12  hours  of  the 
scenario  in  much  faster  than  real  time.  However,  some 
additional  work  is  required  to  ensure  that  the  system  maintains 
this  performance  for  the  full  24  hours  of  the  scenario. 

Assessment  of  the  aggregation  algorithms  by  TNO  and 
ORIGIN,  using  a  Netherlands  national  scenario,  have  shown  an 
excellent  performance  in  aggregating  up  to  regiment  level.  The 
aggregation  algorithms  have  now  been  integrated  in  the  DFS 
and  initial  tests  with  the  International  Scenario  Script  are 
providing  very  promising  results. 

5  THE  FUTURE  PROGRAMME 

5.1  The  Automated  Fusion  Exercise 

After  the  success  of  the  Advisory  Group’s  Manual  Fusion 
Exercise  in  October  1996  to  validate  the  ESS  Message  File,  it 
was  decided  to  hold  an  Automated  Fusion  Exercise  (AFE)  in 
October  1997  to  test  and  validate  the  complete  DFD  with 
military  analysts. 

The  current  plan  is  to  have  a  number  of  Divisional  All  Source 
Intelligence  Cells  operating  in  parallel,  each  being  fed  with  the 
ESS  messages  in  real  time.  There  will  be  four  or  five  cells, 
each  with  two  military  analysts  and  two  or  more  support 
scientists.  The  first  cell  will  be  the  reference  cell  and  be 
manual  only,  the  next  cell  have  the  ESS  graphical  Message 
Viewer  only  and  the  third  cell  will  have  the  DFS  Prototype 
1.21  with  the  classification,  correlation  and  aggregation 
algorithms.  The  fourth  cell  will  be  similar  to  the  third  but  with 
the  addition  of  MODIF  if  the  latter  can  be  completed  in  time. 
In  addition  there  will  be  fifth  cell  to  demonstrate  Prototype  1.5 
which  incorporates  the  new  fusion  algorithms  currently  being 
developed  by  the  Italians. 

Having  a  number  of  cells  operating  in  parallel  will  not  only 
provide  the  means  to  assess  the  effectiveness  of  the  different 
levels  of  automation  but  also  give  a  degree  of  redundancy. 


This  is  intended  to  be  a  research  exercise  and  its  purpose  is 
two  fold,  firstly,  to  provide  a  fitting  end  to  the  major  part  of 
the  project  and  secondly,  to  provide  the  military  and  technical 
assessments  necessary  to  enable  the  Italian  consortium  to 
complete  an  effective  Prototype  2  in  mid  1998. 

6  CONCLUSIONS 

The  Prototype  2  Event  Simulation  System  has  been  completed 
on  schedule  and  has  provided  an  effective  means  of  producing 
and  decomposing  scenarios,  deploying  simulated  sensors  and 
generating  realistic  ADatP-3  message  streams.  In  addition  the 
key  components  of  the  ESS  are  currently  being  re-created  on  a 
PC  NT  platform  in  order  to  provide  low  cost  training  and 
operational  aids. 

The  ESS  Prototype  2  key  deliverables,  the  Message  File,  the 
Scenario  Ground  Truth  database,  the  Route  database,  the 
Avenues  of  Approach  and  the  Collection  Plan  have  also  been 
completed,  validated  and  delivered. 

The  Data  Fusion  System,  the  heart  of  the  Data  Fusion 
Demonstrator,  is  unfortunately  behind  schedule  due  primarily 
to  contractual  and  political  constraints.  Nevertheless  the 
completion  and  integration  of  the  Prototype  1  DFS  is 
proceeding  apace  and  it  is  anticipated  that  there  will  be  a  stable 
and  effective  version  available  for  the  Automated  Fusion 
Exercise  in  October  1997.  The  evidence  todate  is  that  within 
the  next  few  months  the  DFD  will  provide  the  participating 
nations  with  not  only  a  very  valuable  research  tool  but  also 
with  a  number  of  modules  with  the  potential  for  operational 
use. 
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COMMENT/QUESTION: 

Can  the  system  be  used  for  fusion  at  a  lower  level  for  brigade  or  battle  group? 


AUTHOR/PRESENTER’S  REPLY: 

Yes,  provided  the  ORBAT  contains  the  necessary  information  on  the  structure  of  the  lower  level 
units.  The  aggregation  algorithms  have  been  designed  for  use  at  battleground  level. 
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1.  SUMMARY 

A  new  concept  for  designing  infrared  imaging  systems 
is  under  development.  In  this  concept,  absorbed  long 
wavelength  infrared  light  is  up-converted  to  near  infrared 
emission  through  the  vertical  integration  of  a  p-type 
multiple  quantum  well  infrared  photodetector  (QWIP) 
heterostructure  and  a  light  emitting  diode  (LED).  The 
infrared  (IR)  radiation  is  absorbed  in  the  QWIP  part  of 
the  heterostructure.  The  optically  excited  charge  carriers 
from  the  QWIP  are  used  to  stimulate  near  IR  emission 
from  the  LED  layers.  This  emitted  radiation  can  then 
be  imaged  using  a  commercial,  high  resolution  silicon 
CCD  camera.  Because  of  the  extremely  short  path  that 
the  photoexcited  charge  carrier  traverses  between  the 
QWIP  and  LED  layers,  the  emitted  photon  originates  at 
the  same  spatial  position  as  the  absorbed  photon,  such 
that  there  is  a  one-to-one  correspondence  between  the 
long  wavelength  infrared  image  and  the  near  infrared 
image.  With  this  device,  the  need  for  fabricating 
thousands  of  tiny  pixels  on  the  infrared  array  to  generate 
a  high  resolution  image  is  eliminated.  With  the 
elimination  of  pixels  comes  the  elimination  of  the  need 
for  thousands  of  contacts  to  apply  a  bias  voltage  to  each 
individual  pixel.  Only  two  wires  are  required  to  bias 
this  QWIP  and  LED  heterostructure.  For  a  two  color 
infrared  system,  this  concept  would  require  only  three 
wires  for  the  entire  large  area  imaging  array.  Our 
preliminarary  results  for  a  vertically  integrated  p-type 
GaAs/AlGaAs  QWIP  and  LED  heterostructure  will  be 
discussed. 

2.  INTRODUCTION 

For  high  resolution  infrared  imaging,  focal  plane  arrays 
(FPA)  composed  of  more  than  256x256  individual 
detector  elements  (or  pixels)  are  desired.  For  the  mid-  to 
long  wavelength  infrared  region  (3  to  12  |im),  the 
current  infrared  FPA  technology  is  based  on  HgCdTe  or 
InSb  detector  arrays  indium  bump  bonded  to  a  silicon 
multiplexer  for  signal  readout.  This  hybridization 
scheme  has  several  drawbacks. 

Large  focal  plane  arrays  require  high  uniformity  of 
response  over  a  large  area.  For  InSb  and  HgCdTe  arrays 
this  large  area  material  uniformity  is  difficult  to  obtain 
and  drives  up  the  cost  of  the  imaging  array.  The 
recently  developed  quantum  well  infrared  photodetector 
arrays,  grown  by  molecular  beam  epitaxy  ,  have  the 
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potential  for  lower  materials  cost  and  high  uniformity 
across  a  three  inch  wafer.[l]  However,  even  with  well 
controlled,  low  defect  density  epitaxial  materials,  the 
fabrication  of  the  thousands  of  pixels  in  the  arrays  can 
result  in  processing  induced  non-uniformities.  Another 
drawback  of  hybridized  arrays  is  the  mismatch  between 
the  thermal  coefficient  of  expansion  between  the  silicon 
multiplexer  and  the  IR  detector  array.  This  thermal 
mismatch  can  lead  to  debonding  at  the  indium  bumps 
after  repeated  thermal  cycling. 

To  overcome  these  disadvantages  of  standard  FPAs,  Liu 
et  al.[2]  proposed  a  simple  and  innovative  concept  in 
which  a  quantum  well  infrared  photodetector  is  in  series 
with  a  light  emitting  diode  to  upconvert  the  long 
wavelength  infrared  photons  absorbed  by  the  QWIP  into 
near  IR  photons  emitted  by  the  LED.  The  near  IR 
emission  serves  as  an  optical  readout  of  the  longer 
wavelength  IR  image.  The  near  IR  optical  output  can 
then  be  captured  with  a  standard  silicon  CCD  camera  to 
produce  the  desired  visible  image. 

3.  UPCONVERSION  PROCESS 
The  fundamental  processes  of  the  vertically  integrated 
QWIP  and  LED,  mid  to  near  infrared  wavelength  up- 
conversion  device,  are  shown  in  Fig.  1.  There  are 
several  different  epitaxial  layers  in  this  device.  These 
thin  layers  (40A  to  500A)  are  grown  by  molecular  beam 
epitaxy  on  a  GaAs  substrate.  The  GaAs  and  InGaAs 
layers  of  the  LED  are  grown  directly  on  top  of  the 
multiple  GaAs  and  AlGaAs  layers  of  the  QWIP  during 
the  same  growth  run.  Figure  1  shows  the  conduction 
and  valence  band  edge  profile  of  these  layers  using  a  p- 
type  QWIP.  The  p-type  QWIP  is  designed  to  absorb 
mid-infrared  light  which  photo-excites  a  charge  carrier 
(hole)  from  the  confined  ground  state  of  the  GaAs 
valence  band  quantum  well  to  a  state  above  the  potential 
energy  barrier  of  the  wider  bandgap  AlGaAs  layers. 

Under  an  applied  forward  bias,  the  photo-excited  hole 
moves  to  the  LED  layers  and  is  collected  in  the  bottom 
of  the  InGaAs  valence  band  quantum  well.  Electrons 
from  the  top  n+  contact  fall  into  the  InGaAs  conduction 
band  quantum  well  and  recombine  with  the  confined 
holes,  thereby  emitting  a  near  infrared  photon.  The 
intensity  of  the  near  IR  emission  is  directly  related  to 
the  intensity  of  infrared  radiation  absorbed.  These 
devices  have  demonstrated  100%  internal  conversion 
efficiency  of  the  injected  hole  to  near  infrared  photon.  [2] 
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Fig.1  Energy  band  edge  profile  of  a  p-type  QWIP-LED  heterostructure  for  optical  upconversion. 


Only  two  contacts  are  needed  for  this  device:  one  to  the 
heavily  n-doped  LED  contact  layer  and  one  to  the 
heavily  p-doped  QWIP  emitter.  This  is  a  remarkable 
change  from  the  number  of  electrical  contacts  required 
on  a  hybridized  imaging  array. 

4.  NORMAL  INCIDENCE  ABSORPTION 
The  quantum  well  in  the  photodetector  heterostructure 
could  be  doped  with  either  donor  atoms  (n-type)  or 
acceptors  (p-type)  and  still  be  successfully  integrated  to 
the  LED.  In  fact,  the  initial  single  element  QWIP-LED 
device  reported  by  Liu  et  al.fl]  was  based  on  a  n-type 
QWIP.  The  advantage  of  a  p-type  GaAs/AlGaAs  QWIP 
is  that  infrared  radiation  perpendicular  to  the  epitaxial 
layers  can  be  absorbed  in  the  quantum  wells  formed  in 
the  GaAs  valence  band.  This  is  not  the  case  for  n-type 
quantum  wells  in  the  GaAs  conduction  band.  Because 
of  the  isotropic  and  parabolic  nature  of  the  GaAs 
conduction  band,  the  confined  electrons  in  the  quantum 
wells  can  only  couple  to  radiation  with  a  polarization 
component  in  the  confinement  direction,  i.e.  normal 
incidence  absorption  is  forbidden  by  the  dipole  selection 
rules. [3]  Therefore,  n-type  QWIPs  require  diffraction 
gratings  to  be  fabricated  on  the  photodetector  elements 
to  bend  the  light  so  that  it  passes  at  an  angle  through 
the  quantum  well  layers.  Szmulowicz  and  Brown  [4] 
have  shown  that  the  complex  nature  of  the  GaAs 
valence  band  allows  a  p-type  quantum  well  to  efficiently 
absorption  of  radiation  perpendicular  to  the  plane  of  the 
well. 


5.  P-TYPE  QWIP  DESIGN 

For  QWIPs,  the  preferred  heterostructure  design  utilizes 
a  bound-to-continuum  optical  transition.  In  a  bound-to- 
continuum  QWIP,  the  confined  charge  carrier  is  photo- 
excited  from  the  quantum  well  ground  state  to  an 
extended  state  above  the  potential  energy  barrier,  i.e.  in 
the  continuum.  In  this  case,  the  photo-excited  charge 
carrier  is  free  to  move  through  the  AlGaAs  layers  under 
an  applied  voltage.  For  a  p-type  GaAs/ AlGaAs  QWIP, 
there  are  several  different  final  states  that  could  be  used. 
The  calculated  optical  oscillator  strengths  for  various 
possible  final  hole  states  are  shown  in  figure  2.  The 
optical  transition  with  the  largest  oscillator  strength  for 
normal  incidence  absorption  is  the  transition  from  the 
heavy-hole  ground  state  (HH1)  to  the  second  light-hole 
(LH2)  state.  The  HH1  to  LH2  transition  provides  the 
largest  optical  absorption  at  the  center  of  the  Brillouin 
zone,  where  the  majority  of  the  confined  holes  reside. 

The  infrared  absorption  band  of  a  QWIP  is  determined 
by  the  energy  separation  between  the  initial  and  final 
states  of  the  quantum  well.  The  two  parameters  which 
control  this  energy  separation  are  the  quantum  well 
width,  and  depth  (or  AlxGai_xAs  composition).  For  a 
bound-to-continuum  QWIP,  care  should  be  taken  to 
design  the  final  extended  state  just  above  the  top  of  the 
potential  energy  barrier.  The  closer  the  extended  state  is 
to  the  top  of  the  quantum  well,  the  stronger  the  optical 
absorption  will  be.  Figure  3  demonstrates  the  effect  of 
changes  in  quantum  well  width  on  the  calculated  absorp¬ 
tion  spectrum  intensity  and  cut-off  wavelength. 


38-3 


With  our  advanced  theoretical  model,  the  p-type  QWIP 
heterostructure  can  thus  be  designed  to  give  maximum 
photoresponse  in  a  specific  wavelength  band  of  interest. 
Utilizing  the  QWIP  designs  of  Fig.  3,  a  series  of  p-type 
QWIPs  were  grown  by  molecular  beam  epitaxy  for  our 
studies. [5]  The  measured  photoresponse  spectrum,  at 
normal  incidence,  for  one  of  these  samples  is  shown  in 
figure  4.  For  a  p-type  QWIP  with  30%  aluminum  in 
the  barrier  layers  and  a  well  width  of  40A,  the  peak 
photoresponse  is  at  8.5  p.m  and  the  cut-off  wavelength 
is  9.5  pm.  The  measured  photoresponse  spectrum  is  in 
excellent  agreement  with  the  calculated  absorption 
spectrum. 

6.  LED  DESIGN 

The  active  region  of  the  light  emitting  diode,  shown  in 
Fig.  1,  is  very  similar  to  that  of  a  high  performance 
quantum  well  laser.  The  advantage  of  using  an  InGaAs 
well  in  the  active  region  is  that  the  emitted  photons 
will  not  be  absorbed  in  the  wider  bandgap  GaAs  and 
AlGaAs  layers.  A  low  indium  content  InxGai_xAs 
layer  is  used  to  avoid  large  strains  caused  by  the  lattice 
constant  mismatch  between  GaAs  and  InxGai_xAs  with 
increasing  indium.  A  typical  LED  active  region  would 
consist  of  a  90A  wide  quantum  well  composed  of 
Ino.1Gao.9As  and  surrounded  by  GaAs.  An  LED  of 
this  design  has  a  measured  output  wavelength  of  875 
nm. 

7.  PIXELLESS  IMAGING  ISSUES 

In  standard  infrared  imaging  arrays,  pixels  are  required  to 
prevent  optical  and  electrical  cross-talk  between  adjacent 
areas  on  the  array.  However,  the  QWIP-LED  device 
thickness  is  much  smaller  than  typical  alloyed  infrared 
detector  materials,  i.e.  <2  pm  versus  6  pm.  The 
combination  of  the  thin  device  structure  with  the  optical 
readout  capability  is  what  provides  the  opportunity  for 
pixelless  imaging  with  a  QWIP-LED  device. 

There  are  mainly  two  key  requirements  for  upconverting 
a  long  wavelength  infrared  image  into  a  LED  emission 
image  with  negligible  distortion,  smearing,  and  cross¬ 
talk.  First,  the  thickness  of  the  entire  active  region  of 
the  QWIP-LED  structure  must  be  less  than  the  infrared 
wavelengths  to  be  imaged.  This  requirement  is  based 
on  the  diffraction  limit  in  optical  elements. 

Second,  the  charge  carrier  photo-excited  in  the  detector 
layers  must  reach  the  LED  active  region  with  negligible 
motion  perpendicular  to  the  electric  field  of  the  applied 
bias.  Again,  considering  the  diffraction  limit  of  the 
device,  a  perpendicular  motion  less  than  the  infrared 
wavelength  will  have  negligible  effects.  The  photo- 
excited  charge  carrier  from  the  QWIP  only  has  to 
traverse  a  distance  of  about  0.07  pm  before  reaching  the 
InGaAs  well.  The  carrier  diffusion  lengths  are  very 
short  in  this  device,  on  the  order  of  0.1  pm. 


So,  in  principle  the  QWIP-LED  architecture  should  lend 
itself  to  large  area  imaging  without  the  need  for  the 
fabrication  of  individual  detector  elements.  And  without 
the  need  to  form  electrical  interconnects  to  thousands  of 
individual  elements. 

8.  QWIP-LED  DEVICE  PERFORMANCE 

To  date,  several  tests  of  this  pixelless  imaging  concept 
have  been  made.  Allard  et  al. [6]  performed  these  tests 
using  a  p-type  QWIP-LED  heterostructure  grown  by 
molecular  beam  epitaxy  on  a  semi-insulating  GaAs 
substrate.  Details  of  the  epitaxial  layer  thicknesses  and 
compositions  used  in  the  device  are  given  in  Ref.  6. 
For  testing  the  imaging  characteristics,  a  large  area 
device  was  fabricated  using  standard  GaAs  lithography 
and  processing  techniques.  A  3mm  by  4  mm  mesa  was 
etched  to  uncover  a  portion  of  the  buried  bottom 
contact.  A  metallized  ring  contact  was  deposited  on  the 
top  of  the  mesa,  as  well  as  a  metallized  pad  on  the 
bottom  contact  layer.  Therefore,  only  two  wires  were 
needed  to  operate  this  large  area  device.  A  schematic  of 
the  wavelength  upconversion  device  geometry  is  shown 
in  Fig.  5.  [Please  note  that  this  drawing  is  not  to  scale. 
The  device  thickness  (-2  pm)  is  actually  much  smaller 
than  the  device  length  (4  mm).] 

Long  wavelength  QWIPs  need  to  be  cooled  to  cryogenic 
temperatures  to  operate  efficiently.  Therefore,  the 
packaged  QWIP-LED  slab  "array"  was  cooled  to  80K 
using  a  liquid  nitrogen  optical  cryostat.  The  reported 
applied  bias  used  for  testing  was  -3.0  volts  between  the 
top  and  bottom  contacts. 

Two  imaging  tests  have  been  made  with  this  pixelless 
QWIP-LED  device  in  combination  with  a  silicon  CCD 
camera.  The  first  test  used  a  low  intensity  CO2  laser 
beam  focussed  to  one  spot  on  the  upconversion  device. 
This  test  was  to  check  for  any  optical  cross-talk  in  the 
large  area,  pixelless  device,  i.e.  will  it  really  image 
without  pixels?.  A  single  bright  spot,  that  was  the  size 
of  the  mid-infrared  laser  spot,  was  imaged  off  the 
rectangular  surface  of  the  QWIP-LED  "array"  by  the  Si 
CCD  camera.  No  secondary  emission  of  light  from  the 
areas  surrounding  the  laser  spot  was  observed.  Liu  et  al 
[2]  also  showed  that  the  LED  output  power  increased  as 
the  input  CO2  laser  power  was  increased. 

The  second  test  involved  imaging  the  output  of  a 
1000K  blackbody-like  source  that  had  a  1  mm  diameter 
aperature  in  front  of  it.  Results  from  this  test  showed 
that  the  emitted  image  was  a  circular  spot  with  about  a 
1mm  diameter  as  expected.  However,  the  spot  imaged 
from  the  LED  emission  was  -150  pm  smaller  than  the 
diameter  of  the  infrared  spot.  This  change  in  the 
emitted  versus  absorbed  beam  diameters  is  believed  to 
be  an  effect  of  a  reflection  within  the  device's  total 
thickness.  For  this  test  sample,  the  main  contributor  to 
the  packaged  device  thickness  was  the  150  micrometers 
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of  the  thinned  GaAs  substrate.  The  recent  theoretical 
work  of  Ryzhii  et  al. [7]  found  that  the  image  transfer 
from  the  infrared  generated  current  into  the  near  infrared 
emission  should  be  very  close  to  one-to-one  with  no 
image  smearing  during  the  upconversion.  Therefore, 
further  thinning  or  removal  of  the  GaAs  substrate 
should  result  in  improved  image  transfer  from  the  mid- 
infrared  scene  to  the  Si  CCD  camera. 

9.  MULTI-SPECTRAL  SENSING 

The  QWIP-LED  concept  can  be  applied  to  other 
quantum  well  material  systems  and  other  GaAs/AlGaAs 
quantum  well  designs,  i.e.  with  different  AlGaAs  alloy 
compositions  and  well  widths,  in  addition  to  the 
heterostructures  discussed  in  this  paper.  This  flexibility 
allows  us  to  design  QWIP  and  LED  combinations  for 
more  than  one  spectral  band.  The  tremendous  advantage 
of  needing  only  two  contacts  for  the  whole  array,  versus 
the  thousands  of  electrical  contacts  on  a  hybridized 
array,  becomes  even  more  obvious  for  a  multi-color 
sensor  array.  For  a  two-color  QWIP-LED  array,  we 
need  only  design  a  second  QWIP-LED  combination  that 
absorbs  in  a  second  infrared  band  and  has  a  LED  layer 
that  emits  a  different  near  IR  wavelength.  Both  sets  of 
QWIP-LED  layers  would  be  sequentially  grown  during 
the  same  molecular  beam  epitaxy  growth  run.  Three 
color  silicon  CCD  cameras  are  already  available,  so  the 
two  separate  near  IR  or  visible  emissions  could  be 
captured  and  fused  into  one  visible  image.  Or,  the  two 
separate  emissions  could  be  used  for  other  image 
processing  needs.  This  double  QWIP-LED  structure 
would  require  at  most  three  wires  for  the  whole  array. 

10.  CONCLUSIONS 

A  new  concept  for  infrared  imaging  devices  has  been 
presented.  By  sequentially  growing  light  emitting  diode 
layers  on  top  of  quantum  well  infrared  photodetector 
layers,  a  pixelless  imaging  device  with  an  optical 
readout  is  created.  Since  the  active  region  of  the  device 
is  very  thin,  there  is  a  direct  mapping  of  the  spatial  and 
intensity  information  in  the  infrared  scene  into  a  near  IR 
image.  Without  the  need  for  fabricating  and  bump 
bonding  tens  of  thousands  of  tiny  pixels,  a  much 
simpler  and  lower  cost  infrared  imaging  array  can  be 
created. 

These  pixelless  QWIP-LED  imaging  devices  are  only  in 
the  early  stages  of  development.  The  full  capabilities  of 
these  structures  have  not  yet  been  explored.  Further 
testing  on  improved  p-type  QWIP-LED  structures  is 
planned.  The  elegant  simplicity  of  these  integrated 
devices,  especially  for  multi-spectral  imaging,  makes 
this  a  technology  capable  of  creating  a  new  generation 
of  infrared  imaging  arrays. 
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Fig.  2  Squared  momentum  matrix  elements  in  the  x-direction  for  intersubband  transitions 
originating  with  the  HH1  band  as  a  function  of  kj|  along  the  (11)  direction  for  a  50A 
GaAs/Alo.3oGao.7As  quantum  well  (a.u.  =  0.529177  A). 


Fig.  3  Comparison  of  the  calculated  bound-to-continuum  absorption  spectra  of  three  p- 
type  quantum  wells,  for  infrared  radiation  at  normal  incidence. 
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Fig.  4  The  measured  photoresponse  spectrum  at  normal  incidence  of  a  p-type  (beryllium 
doped)  GaAs/Alo.3Gao.7As  QWIP  with  a  well  width  of  40A. 


Fig.  5  Schematic  of  the  wavelength  up-conversion  device  geometry  depicting  the  large 
mesa  of  the  QWIP-LED  device  in  combination  with  a  silicon  CCD  imager. 
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COMMENT/QUESTION: 

Is  D*  compromised  by  addition  of  LED  interface  and  what  D*  performance  is  likely? 


AUTHOR/PRESENTER’S  REPLY: 

LED  interface  doesn’t  compromise  D*. 

D*  is  about  x2,  x3,  less  (worse  than)  HgCd  devices. 
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SUMMARY 

There  exists  many  nonlinear  frequency  conversion 
materials  today  which  provide  access  to  wavelength 
bands  not  previously  possible  by  other  means 
(solid  state,  gas,  or  semiconductor  lasers).  Of 
particular  interest  are  optical  sources  for  the  3-5  pm 
and  the  8-12  pm  wavelength  bands.  Operation  in 
these  optical  transmission  bands  is  very 
advantageous  due  to  the  large  optical  absorption  of 
important  chemical  and  biological  materials  and  the 
low  absorption  of  the  atmosphere.  A  brief 
introduction  of  the  uses  of  optical  remote  sensing 
and  the  unique  properties  of  nonlinear  materials  for 
laser  frequency  conversion  are  described. 

1  INTRODUCTION 

Many  future  optical  remote  sensing  and 
surveillance  systems  will  benefit  from  the  use  of 
nonlinear  frequency  converters  to  provide  high 
optical  power  at  a  diversity  of  wavelengths.  These 
materials  and  laser  sources  are  currently  under 
development  for  applications  such  as  pollution 
monitoring,  urban  and  agricultural  planning, 
weather  forecasting,  industrial  process  monitoring, 
biological  pollution  detection,  wind  shear  detection, 
and  chemical  agent  detection.  Many  current  laser 
systems  do  not  possess  wavelength  tunability  and 
are  often  limited  in  their  usefulness  for  many 
advanced  remote  sensing  applications. 

Accurate  remote  detection  of  biological  and/or 
chemical  materials  requires  the  use  of  these 
frequency  agile  laser  sources  capable  of  high  power 
optical  output  in  multiple  optical  wavelength 
bands.  This  requirement  stems  from  the  diversity 
of  absorption  peaks  in  these  materials  and  the 
existence  of  atmospheric  absorption  bands.  Optical 
source  requirements  can  be  fulfilled  with  the  use  of 
nonlinear  frequency  conversion  materials.  Highly 
efficient  frequency  converters  are  now  capable  of 
reaching  many  of  the  important  optical  bands 


including  ultraviolet,  visible,  near  infrared,  3-5  pm, 
and  8-12  pm  with  sufficient  output  power. 

2  OPTICAL  REMOTE  SENSING 

Remote  sensing  has  been  a  topic  of  interest  for 
many  years.  Modern  examples  include  the  use  of 
radars  for  image  mapping  from  satellites  and  ground 
based  systems.1"3  These  systems  rely  on  active  or 
passive  detection  of  radiation  from  a  target  of 
interest  and  the  interpretation  of  this  detected 
signal.  In  the  case  of  microwave  remote  sensing, 
information  such  as  weather,  foliage  cover,  and 
water  temperature  can  be  gathered.2  For  optical 
remote  sensing  (often  called  LIDAR  for  Light 
Detection  And  Ranging)  more  detailed  information 
of  the  molecular  composition  of  a  target  can  be 
determined  from  any  one  of  many  measurement 
techniques. 

These  techniques  range  from  the  use  of  light 
scattering  from  the  medium  which  returns  and  is 
detected  with  no  shift  in  wavelength  (Rayleigh  and 
Mie  scattering)  to  techniques  which  rely  on  small 
wavelength  shifts  in  the  detected  signal  (Raman 
scattering).  Other  techniques  are  based  on  the 
optical  absorption  spectrum  of  a  chemical  or 
biological  sample  (Differential  Absorption  and 
Scattering  (DAS),  and  Differential  Absorption  in 
LIDAR  (DIAL)). 1 

The  technique  which  has  gained  the  most  favor 
recently  is  differential  optical  absorption.  DIAL 
makes  use  of  two  optical  wavelengths.  One  of 
these  wavelengths  is  centered  at  the  chemical  of 
interest’s  optical  absorption  peak.  The  second 
wavelength,  the  reference  wavelength,  is  typical 
chosen  to  be  off  the  peak  in  optical  absorption  and 
is  used  to  help  reduce  noise  and  increase  sensitivity. 
Due  to  the  tunability  needed  to  make  such  a 
technique  possible,  DIAL  systems  are  ideal 
candidates  for  wavelength  agile  laser  sources. 
Typically,  a  DIAL  system  consists  of  two  major 
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subsystems;  (1)  the  optical  source  system  to 
generate  a  probe  beam  and,  (2)  the  collection 
system  (Figure  1). 


Laser 


Figure  1 .  A  typical  LIDAR  system  consisting  of  a 
transmit  and  receive  subsystems. 

The  probe  beam  is  generated  from  a  well  controlled 
high  power  pulsed  laser  system.  Diagnostic 
measurements  are  made  of  important  pulse 
characteristics  such  as  energy,  timing  and 
spectroscopic  information.  The  optical  beam  is 
then  sent  out  into  the  atmosphere  though  a  lens 
system  for  beam  shaping  and  collimation.  When 
the  reflected  light  returns  from  the  target,  a 
collection  system  consisting  of  lenses  and  mirrors, 
directs  and  focuses  the  optical  radiation  into  a 
spectrometer  system  and  detector.  The  detected 
signal  is  then  sent  to  a  signal  processor  for 
interpretation,  data  reduction,  and  readout.  With 
this  fairly  simple  setup,  a  mapping  of  the 
molecules  between  the  LIDAR  system  and  the 
target  can  be  determined. 

Methods  such  as  DIAL  are  useful  because  they  are 
capable  of  remotely  determining  properties  of 
materials  and  aerosols.  It  can  determine  if  a 
chemical  is  present  above  a  certain  concentration 
threshold  or  even  determine  its  absolute 
concentration.  Thermal  structure  of  clouds  or 
exhaust  plumes  can  be  monitored  due  to  the 
temperature  variation  of  absorption  characteristics 
(both  static  and  dynamic).  Finally,  it  can  determine 
a  chemical  species  presence  due  to  its  unique 
spectral  fingerprint. 

High  performance  optical  remote  sensing  systems 
also  require  high  sensitivity  and  resolution,  large 
standoff  range,  and  wavelength  tunability  to  look  at 
many  types  of  materials.  These  requirements  are 
demanding  on  both  the  detection  hardware  and  the 
laser  probe  source.  For  example,  lasers  are  required 
to  have  large  output  powers  to  increase  the  range 


and  sensitivity  of  the  LIDAR  system.  They  should 
also  have  a  narrow  optical  bandwidth  and  short 
pulse  duration  for  increased  sensitivity  and 
resolution.  High  pulse  repetition  rates  are  needed 
for  noise  reduction  and  averaging. 

Such  stringent  requirements  on  the  optical  source 
also  imposes  severe  requirements  on  the  optical 
source  materials.  As  next  generation  systems  are 
designed  and  built,  nonlinear  materials  are  being 
increasingly  sought  to  help  provide  such  an  ideal 
laser  source.  The  advantages  of  designing  a  source 
based  on  nonlinear  materials  are;  (1)  solid  state 
materials  offer  high  rugginess,  high  efficiency  and 
low  part  count,  (2)  a  large  operating  wavelength 
spectrum  covering  visible,  mid-infrared,  and  far- 
infrared  spectral  windows,  (3)  continous  wavelength 
tunability  over  several  micron,  (4)  high 
performance  optical  beam  properties  (high  output 
power,  large  repetition  rates,  and  short  pulses),  (5) 
decreased  cost. 

3  FREQUENCY  CONVERSION  MATERIALS 

The  use  of  nonlinear  crystals  for  frequency 
conversion  followed  shortly  after  the  discovery  of 
the  first  laser.  Whether  the  need  is  to  gain  access 
to  wavelengths  unobtainable  from  the  current  laser 
technology  or  to  increase  the  ability  of  an  optical 
system  for  frequency  tunability,  the  need  to  shift 
optical  power  from  one  wavelength  to  another  has 
always  been  important. 

The  earliest  nonlinear  materials  were  often 
inefficient,  expensive,  and  riddled  with  defects. 
Over  the  last  thirty  years,  many  material 
improvements  and  newer  materials  with  better 
properties  have  been  realized.  These  include 
materials  for  frequency  conversion  in  the 
ultraviolet,  visible,  and  infrared.  Because  of  the 
wide  range  of  application  and  requirements,  the 
material  is  usually  optimized  for  a  specific 
application  and  pump  source. 

Frequency  conversion  occurs  in  a  dielectric  medium 
when  a  high  energy  optical  field  is  incident  upon  a 
material  causing  a  nonlinear  response  of  the 
material’s  refractive  index.  This  change  allows 
coupling  of  the  electric  field  (pump  beam)  at  one 
wavelength  to  electric  fields  at  other  wavelengths 
(signal  and  idler  beams).  This  comes  with  the 
requirement  that  energy  is  conserved  between  the 
pump,  signal  and  idler  beams.  Also,  the  process 
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will  only  occur  with  high  efficiency  if  the  relative 
phase  of  all  the  beams  remains  nearly  zero. 
Therefore,  two  requirements  must  be  satisfied, 
namely,  conservation  of  energy,  expressed  as: 

(Op  =  COs  +  COi  (1) 

where  cop,  tos,  and  C0j  are  the  wavelengths  of  the 
pump,  signal  and  idler  beams  respectively,  and 
conservation  of  phase  (or  momentum)  which  is 
expressed  as: 


This  can  force  the  use  of  a  lower  nonlinear 
coefficient  since  the  largest  nonlinear  tensor  does 
not  always  lie  in  the  phase  matching  direction. 
Even  with  this  restriction,  many  high  power 
systems  are  based  on  these  materials.  Figure  3 
shows  a  graph  of  many  well  developed  nonlinear 
materials,  for  infrared  applications  and  their 
transmission  range  and  figure  of  merit.  Two  of  the 
most  important  birefringent  materials  for  this 
wavelength  band,  ZnGeP2  and  GaSe,  are  currently 
being  developed  for  optical  sources. 


kp  =  ks  +  kj  (2) 

where  kp,  ks,  and  kj  are  the  wave  vectors  of  the 
pump,  signal,  and  idler  beams  respectively.  In  the 
case  of  nonlinear  frequency  conversion,  the  second 
condition  (Equation  2)  is  the  more  restrictive.  In 
order  to  have  the  phase  of  different  wavelengths 
beams  remain  the  same,  the  velocity  of  the  waves 
must  be  identical.  One  method  to  accomplish  this 
condition  in  a  material  is  depicted  in  Figure  2.  In 
this  case,  the  material’s  natural  birefringence  or 
difference  in  refractive  index  for  different  crystalline 
directions  is  used  to  compensate  for  the  different 
propagation  speed  of  each  wavelength.  The 
directions  are  known  as  the  ordinary  and 
extraordinary  direction.  This  method  of  optical 
phase  compensation  is  known  as  birefringent 
phasematching.  Unfortunately,  this  technique 
requires  very  strict  control  over  the  direction  of 
propagation  in  the  crystal,  necessitating  accurate 
polishing  and  cutting  techniques. 


Figure  2.  Birefringent  phasematching  requires  a 
material  to  be  anisotropic  and  limits  the  direction 
of  propagation  of  electric  fields  for  efficient 
frequency  conversion. 


10 

10 

10 

10 

10 

o  5  10  15  20  25 

Transparency  Range  -  (microns) 


ZnGeP 


■  CdGeAs 

.  GaSe 
-AgGaSe 


AgGaS 


LiNbO 


Figure  3.  Figure  of  merit  and  transmission  range 
for  many  common  nonlinear  frequency  conversion 
materials. 


ZnGeP2  belongs  to  the  chalcopyrite  family  of 
crystals.  As  it  can  be  seen  in  figure  4,  ZnGeP2has 
a  very  large  optical  gain  making  it  very  useful  for 
high  power  optical  sources.  One  problem  with  this 
material,  however,  is  the  observation  of  a  crystal 
defect  absorption  band  near  2  pm  (see  Figure  5). 


Figure  4.  Optical  Gain  in  ZnGeP2  (From  ref.  5) 
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This  absorption  band  has  recently  been  the  topic  of 
intense  study.  Reductions  in  the  optical  absorption 
through  better  crystal  growth  techniques  and  a 
better  understanding  of  the  defect  structure  have 
been  achieved. 
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[—  o^xVarized  —  frpolarizad 


Figure  5.  Optical  absorption  in  ZnGeP2  crystals 
grown  by  the  horizontal  Bridgeman  method. 

Gallium  Selenide  (GaSe)  is  another  nonlinear 
material  which  has  recently  received  attention. 

This  material  has  a  very  large  transparency  range  of 
approximately  1  pm  to  18  pm  (Figure  6),  making 
it  an  extremely  interesting  material  for  applications 
in  the  mid-  and  far-infrared. 8  The  measured 
nonlinear  coefficient  of  ~55  pm/V  for  GaSe  is 
adequate  for  most  application.  This  material, 
because  of  its  mica-like  structure,  is  difficult  to 
process.  Cutting  and  polishing  is  of  particular 
difficulty  in  this  material  because  the  crystal’s 
tendency  to  cleave  in  thin  sheets  only  along  the 
[0001]  crystalline  direction.  Recent  attempts  at 
doping  and  optimizing  the  growth  parameters  has 
reduced  this  problem.  Even  so,  many  optical 
systems  have  been  designed  using  GaSe  resulting 
in  very  functional  sources  with  optical  wavelength 
outputs  over  a  3  pm  to  18  pm  range. 


Figure  5.  Optical  transmission  curve  for  GaSe. 


Another  methodology  for  achieving  efficient 
frequency  conversion  is  quasi-phase  matching 
(QPM).  Although  this  type  of  phase  matching  was 
first  described  in  1962,  it  wasn’t  until  recently  that 
materials  and  processes  were  developed  to  take  full 
advantage  of  this  technique.6  In  this  method,  a 
grating  is  used  to  compensate  for  the  mismatch  of 
phase  in  the  propagating  wavelengths  instead  of 
crystal  birefringence  (Figure  7).  This  is 
accomplished  by  periodically  inverting  the  direction 
of  the  nonlinear  tensor  so  that  as  the  waves  move 
out  of  phase  the  next  periodic  section  brings  them 
back  into  phase.  This  is  equivalent  to  adding  a 
grating  wave  vector  of  magnitude  8K. 


kp 

- 


ks  kj  K 


Figure  7.  Quasi-phase  matching  uses  a  grating  to 
conserve  momentum  during  frequency  conversion. 
The  grating  is  formed  by  periodically  inverting  the 
nonlinear  tensor. 

Quasi-phase  matching  is  a  very  powerful  technique. 
The  direction  of  the  optical  axis  can  now  be  in  the 
direction  of  the  highest  nonlinearity  .  The  crystal 
can  also  be  customized  for  specific  applications  or 
wavelength  conversion  through  external  processing 
after  crystal  growth.  This  technique  does  however 
require  a  material  whose  nonlinear  tensor  can  be 
altered  in  a  periodic  fashion.  Materials  like 
ferroelectrics  have  the  property  of  being  periodically 
poled  under  a  large  field.  This  field  causes  the 
ferroelectric  domains  to  reverse  and  hence  reverse 
the  nonlinear  tensor.  For  example,  QPM  has  been 
successfully  demonstrated  in  multi-grating 
periodically  poled  lithium  niobate  (PPLN).  The 
periodicity  of  the  ferroelectric  domains  can  be  easily 
controlled  through  photolithography  to  achieve  any 
grating  spacing  to  phase  match  the  pump,  signal 
and  idler  optical  waves.  Large  tuning  ranges  ( 1 .4 
pm  -  4.8  pm)  and  powers  (>3  Watts)  have  already 
been  demonstrated  using  this  technique  (Figure  8). 7 
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Grating  Period  (pm) 

Figure  8.  (a)  Typical  optical  parametric  oscillator 
configuration  for  a  multi-grating  Periodically  Poled 
Lithium  Niobate  crystal  and  (b)  corresponding 
tuning  curve  showing  a  large  wavelength  tuning 
range. 

4  CONCLUSIONS 

Nonlinear  materials  exist  today,  and  others  are 
being  developed,  capable  of  highly  efficient  laser 
frequency  conversion.  Optical  sources  based  on 
these  materials  are  ideal  for  insertion  into  remote 
sensing  systems.  By  providing  wavelength 
tunability  over  large  optical  spectral  bands, 
improved  sensing  techniques  are  possible  for 
systems  with  better  sensitivity  and  operational 
performance.  Materials  like  ZGP  and  GaSe  have 
established  themselves  as  high  performance 
materials,  what  remains  is  the  transition  of  this 
technology  to  new  application  like  remote  sensing. 
Other  materials  and  technique,  periodically  poled 
lithium  niobate  for  example,  are  still  in  their 
infancy  but  are  rapidly  making  an  impact  on  this 
technology. 
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•  Introduction  to  Remote  Sensing 

•  Requirements  for  Optical  Remote  Sensing 

•  Laser  /  Optical  Parametric  Oscillator  Fundamentals 

•  Nonlinear  Crystals  for  Frequency  Conversion 

ZnGeP2 

Periodically  Poled  Lithium  Niobate 

GaSe 

•  Conclusions 


Why  Do  We  Need  Remote  Sensing? 


Example  of  RADAR  Remote  Sensing 


Environmental  Chemical 

•Urban  Planning  'Pollution  Monitoring 

•Weather  Forecasting  *Chemical  Agent  Detection 

•Agricultural  Planning  'Process  Monitoring 


Rinlogiral 

•  Biological  Pollution  (Ride  Tide) 

•  Biological  warfare  agent  detection 


Typical  Optical  Remote  Sensing  System 
(LIDAR) 


Laser 


Optical  Remote  Sensing  Measurement  Capabilities 


•  Concentration  measurements 

•  Thermal  structural,  and  dynamic  properties 

•  Threshold  detection 

•  Plume  dispersal 

•  Spectral  fingerprinting 
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Laser  Remote  Sensing  Techniques 


DIAL  for  Remote  Sensing  of  Chemical  Species 


•  Rayliegh  Scattering 

•  Mie  Scattering 

•  Raman  Scattering 

•  Resonance 

•  Flourescence 

•  Absorption 

•  Differential  Absorption  (DIAL) 


•Very  Sensitive  detection  of  chemical  species 

•  Good  Spatial  Resolution 

•  Requires  two  wavelength  (On-peak,  Off-Peak) 


Laser  Source  Requirements  for  LIDAR 

•  High  Peak  Output  Power  (>1  MW  Peak) 

•  Narrow  Bandwidth  (<2  nm) 

•  Frequency  Agility  (>50  nm) 

•  Output  Power  and  Wavelength  Stability 

•  Narrow  Optical  Pulses  (<100  ns) 

•  High  Repetition  Rate  (>1  kHz) 


Current  Laser  Types  for  Remote  Sensing 


•  Nitrogen  Lasers  (Raman  Backscattering) 

•  Q-switched  Ruby  Laser 

•  Quadrupled  Nd-YAG 

•  Carbon  Dioxide  (C02) 

•  Dye  Lasers  (Useful  for  DIAL  Measurements) 

•  Diode  Lasers  (Short  Distances) 


NLO  Crystals  Advantages 

•  Solid  State  (Increased  Reliability) 

•  Large  Tuning  Range  (Many  Microns) 

•  Availability  of  Many  Wavelengths  (Visible-IR) 

•  High  Efficiency 

•  High  Power,  Repetition  Rate,  Narrow  Pulses 

•  Cheap 


Nonlinear  Frequency  Conversion 


COp  =  COs  +  CDi 

l^lq  +  ki 

Energy  and  phase  conservation  eqns. 


Phasematching  Direction 


-g  -» 


Quasi-Phase  Matching 


V- 
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Optical  Parametric  Oscillators 


Broad  Range  of  Bulk  NLO  Materials 


•  Similar  to  Laser 

•  Parametric  Mechanism 

•  Satisfy  Conservation  of  Energy  and  Momentum 

•  Angle  or  Temperature  Tunable 

•  High  Efficiency  Possible 


Optical  Parametric  Oscillator 


Zinc  Germanium  Phosphide  (ZnGePz) 


ZnGeP2  -  The  Material  for  New  Infrared  Lasers 


•  For  Missile  Countermeasures 


*  For  Detection  of  Chem  /  Bio 
Warfare  Agents 


Dual  Use: 

•  Environmental  Monitoring 

•  Wind-Shear  Detection 

•  Laser  Radar 

•  Medical  Lasers 

» Materials  Processing 


Mid  IR  Multigrating  PPLN 


Tuning  from  -1.4  -  4.8  Jim 
Multi  -Watt  Output  Power 
Suffers  from  Photorefractive  Effects 
Low'  Cost  Material 
Uses  1.06  Jim  Laser  Pump 


Gallium  Selenide  (GaSe) 


Other  Applications  for  NLO  Crystals 


Conclusions 


•  Infrared  Countermeasures 

•  Medical  Lasers 

•  Optical  Memories 

•  Guide  Stars 

•  Spectroscopy 


•  Remote  Sensing  is  on  the  Rise 

•  Current  Laser  Sources  Not  Optimal 

•  Nonlinear  Bulk  Crystals  Can  Add  Increased 
Functionality  to  Remote  Sensing  Systems 

•  NLO  Crystals  on  the  Verge  of  Going  Commercial 
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1.  SOMMAIRE 

Les  travaux  entrepris  pour  le  compte  du  Groupe  de 
Coordination  Ddfense  Adrienne  de  la  DRET,  proposent  une 
mdthodologie  d’aide  pour  la  configuration,  l’dvaluation  et 
1’  amdlioration  des  performances  de  systdmes  multi-senseurs 
sol-air  de  type  SACP.  Us  ontdtdconcrdtisdsparlardalisation 
du  logiciel  ROSACE  (Recherche  pour  1’ Optimisation  de 
Systdmes  Anti-adriens  Conception  Evaluation)  ddcrit  dans 
[5]  et  [1 0], 

La  premidre  partie  de  ce  document  prdsente  le  logiciel 
ROSACE  La  seconde  partie  est  consacrde  au  probleme  de 
1’  acheminement  des  informations  au  sein  du  systeme 
multi-senseurs.  En  particulier  un  algorithme  est  prdsentd, 
qui  peimet  de  gdrer  1’  allocation  des  ressources  et  le  transfert 
des  informations  pour  que  la  logique  algorithmique  soit 
respectde. 

^INTRODUCTION 

Les  travaux  entrepris  pour  le  compte  du  Groupe  de 
Coordination  Ddfense  Adrienne  de  la  DRET  proposent 
une  mdthodologie  d’aide  pour  la  configuration,  l’dvalu¬ 
ation  et  1’ amdlioration  des  performances  de  systdmes 
multi-senseurs  sol-air  de  type  SACP.  Ils  ont  dtd 
concretisds  par  la  rdalisation  du  logiciel  ROSACE 
(Recherche  pour  l’Optimisation  de  Systemes,  Anti 
-aeriens  Conception  Evaluation)  ddcrit  dans  [5]  et  [10]. 
La  premiere  partie  de  ce  document  prdsente  le  logiciel 
ROSACE.  La  seconde  partie  est  consacrde  au  probldme 
de  P  acheminement  des  informations  au  sein  du  systdme 
multi-senseurs.  En  particulier,  les  conditions  de  compa- 
tibilite  entre  logique  algorithmique  et  rdseau  de  commu¬ 
nication  physique,  sont  dtudides. 

2.1  Pose  du  probldme 

Un  systeme  multi-senseurs  sol-air  est  constitue  d’un 
ensemble  de  capteurs,  (radars,  camdras  optiques  ou  infra- 
rouges)  rdpartis  sur  une  rdgion  dont  la  surveillance  ou  la 
ddfense  doivent  etre  assurdes.  Des  moyens  de  communi¬ 
cation  (fibres  optiques,  radio,  cables  coaxiaux)  permettent 
1’ acheminement  des  informations  vers  des  centres  de 
decision,  1’ exploitation  de  donndes  d’origine  diffdrente  par 
des  processeurs  de  traitement  Un  systeme  de  ddfense 
comporte  en  outre  des  moyens  de  ddfense  (batteries  de  tir, 
missiles...). 

Le  probleme  de  ropdrationnel  militaire  consiste  dans  un 
premier  temps  a  sdlectionner  les  capteurs  qui  rdpondent  au 


mieux  a  un  cahier  des  charges.  Le  cout,  la  fiabilitd,  les 
performances  des  composants  sont  alors  dtudids  en  fonction 
des  objectifs.  Des  outils  d’aide  au  choix  des  composants  ont 
dtd  ddveloppds  com  me.  par  exemple  l’outil  SADDAA  (Voir 
[4]).  Mais  l’opdrationnel  est  ensuite  confrontd  au  probleme 
de  la  configuration  du  systeme  sur  le  thdatre  des  opdrations, 
c’  est-a-dire  au  positionnement  gdographique  des  capteurs, 
des  moyens  de  ddfense,  a  la  ddfinition  du  rdseau  de 
communication.  En  effet,  son  objectif  est  d’ assurer  une 
bonne  visibilitd  compte  tenue  de  l’environnement  et  en 
particulier  des  masquages  par  le  terrain,  une  bonne  detection, 
une  localisation  suffisamment  prdcise  des  engins  adriens. 
Tout  ceci  doit  etre  pensd  avant  meme  que  le  systeme  ne  soit 
confrontd  a  un  scdnario  rdeL 

Le  scientifique  se  pose  quant  a  lui  la  question  des  traitements 
et  de  leur  amdlioration  pour  l’accomplissement  des  diffd- 
rentes  taches  de  ddtection,  pistage,  classification/denom- 
brement  d’  engins  adriens.  Notamment,  F  apport,  F  interet  et 
le  cout  de  la  fusion  d’  informations  doivent  etre  dtudids.  Mais 
qui  dit  fusion  dit  dchange,  transfert  de  donndes.  Les 
algorithmes  sont  done  dvidemment  contraints  par  les 
possibilitds  de  communication  au  sein  du  systdme. 

L’ objectif  visd  est  Fdlaboration  d’un  logiciel  interactif, 
dvolutif,  et  modulaire  d’  aide  k  la  conception  et  a  F  dvaluation 
d’un  systeme  multi-senseurs  pour  la  surveillance  et  la 
ddfense  anti-adrienne.  II  ne  s’agit  pas  de  mettre  au  point  un 
simulateur  qui  reproduirait  dans  leurs  moindres  ddtails  les 
fonctionnements  des  capteurs  dans  unenvironnementdonne. 
H  s’agit  plutot  de  rechercher  des  critdres  qualifiants  les 
performances  opdrationnelles  globales  d’un  systeme  puis 
d’amdliorer  ces  performances  par  modification  des  para- 
mdtres  du  systdme  ou  de  F  algorithmic. 

Afin  de  ddmontrer  la  faisabilitd  d’un  tel  produit,  certaines 
limitations  sont  imposdes  quant  a  la  varidtd  des  capteurs, 
des  menaces,  des  parametres  d’environnement  Certaines 
simplifications  concemant  les  phdnomenes  physiques  sont 
adoptdes  mais  ndanmoins,  les  situations  envisagdes  con- 
servent  un  caractere  idaliste. 

Les  capteurs  considdnds  sont  des  radars  de  veille,  ou  des 
camdras  infrarouges.  Dans  un  premier  temps,  il  est  proposd 
de  synthdtiser  les  informations  pertinentes  pour  la  speci¬ 
fication  d’une  situation  et  la  simulation  des  missions 
d’  engins  adriens  ennemis.  Les  menaces  sont  des  hdlicopteres, 
avions  ou  missiles.  Ensuite,  il  s’agit  de  rdpertorier  et 
d’  dvaluer  les  algorithmes  de  fusion  qui  accomplissent  les 
taches  de  ddtection,  localisation,  classification  et  pistage  des 
engins  adriens. 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar",  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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SCENARIO 


2.2  L’outil  ROSACE,  outil  mdthodologique 

L’outil  ROSACE  a  dtd  idalisd  pour  repondre  a  ces 
problemes.  C'est  un  logiciel  interactif,  ergonomique,  por¬ 
table  sur  station  Unix  supportant  X- Windows  et  modulable. 
Ce  n’est  pas  un  logiciel  dcrit  selon  les  nonnes  industrielles 
mais  une  maquette.  Dans  son  6tat  actuel,  il  comporte  quatre 
modules : 

-  Le  module  SITUATION  offre  la  possibilitd  de  ddfinir  et 
de  configurer  interactivement  un  systeme  multi-senseurs 
sur  une  region  d’  intdret.  Une  carte  de  la  rdgion  peut  etre 
visualisde  a  1’  dcran.  L’  opdrateur  sdlectionne  les  composants 
du  systeme  (capteurs,  moyens  de  ddfense)  dans  une  base  de 
donndes  piddtablie.  Ces  composants  sont  reprdsentds  a 
f  dcran  par  des  i cones.  D  les  positionne  et  les  oriente  sur  la 
carte.  II  ddfinit  ensuite  un  idseau  de  communications  (des 
moyens  de  communications  type  figurent  dans  la  base  de 
donndes). 

-  Le  module  ANALYSE  propose  une  Evaluation  globale 
des  performances  du  systeme  (c’est-a-dire  inddpendam- 
ment  des  scdnarios  auxquels  le  systeme  sera  confrontd).  Ces 
criteres  comprennent  le  calcul  de  la  visibility  et  de 
F  intervisibility  des  capteurs  avec  prise  en  compte  du 
masquage  par  le  relief  du  terrain,  le  taux  de  detection  d’un 
aeronef  supposd  situd  en  un  point  de  F  espace.  ainsi  que  le 
taux  de  fausse  alarme,  la  prdcision  de  la  localisation,  la 
ldtalitd.  Les  idsultats  sont  repidsentds  graphiquement  de 
maniere  synthdtique.  De  plus  un  partitionnement  de  F  espace 
adrien  surveilld  est  proposd.  Ce  partitionnement  permet 
d’attribuer  la  responsabilitd  d’une  idgion  de  F  espace  il  un 
groupe  de  capteurs. 

Les  rdgions  insuffisamment  surveilldes  ou  protdgdes  appa- 
raissent  alors  et  suggerent  a  F  opdrateur  une  modification  de 
la  situation  done  un  rebouclage  sur  le  module  SITUATION. 
Ce  module  est  le  coeur  du  logiciel  et  en  fait  son  originalitd. 
II  ne  s’agit  pas  d’effectuer  des  traitements  mais  d’dvaluer 
avant  tout  leurs  performances. 

-  Le  module  SCENARIO  permet  de  ddfinir  et  de  visualiser 
interactivement  des  raids  (ensemble  de  trajectoires  d’  avions, 
d’  hdlicopteres  ou  de  missiles)  grace  if  F  impldmentation  d’  un 
dditeur  de  trajectoires  adapt d.  En  outre  le  sednario  peut 
comporter  brouilleurs,  adrosols,  et  diffdrentes  conditions 
atmosphdriques  peuvent  etre  prdcisdes. 

-  Enfin  le  module  SIMULATION  prend  en  compte  la 
probldmatique  temporelle  et  donne  une  id de  du  comport- 
ement  du  systdme  en  dtat  de  fonctionnement  Les  taches  de 
detection,  pistage,  classification  sont  done  simuldes  et 
accomplies  par  des  algoritlimes  de  fusion  qui  exploitent  le 
rdseau  de  communications  pour  amdliorer  le  comportement 
du  systeme. 

ORGANISATION  GENERALE  DU  LOGICIEL  ROSACE 

SITUATION 

Thdatrc 

Capteurs 

Postes  de  ddfense 

Logique  de  fusion 

Communications 


Edition  de  raids 
Brouilleurs 
Conditions  mdtdo 
Adrosols 

ANALYSE 

Couverture  par  capteur 
Ddtection/Fausses  alarmes 
Pidcision  en  localisation 
Ldtalitd _ 

SIMULATION 

Mesures 

Ddtection 

Pistage 

Classification 

Ddnombrement 

OPTIMISATION 

Seuils  capteurs 

Position/oiientatior 

Algorithmic 

i 

A  titre  d’ illustration,  la  figure  1  propose  un  exemple  de 
configuration  d’un  systeme  multi-capteurs  de  veille.  Le 
thdatre  des  opdrations  est  reprdsentd  par  une  image  SPOT. 


Figure  1 :  Configuration  d  un  systdme  sol-air 
Chaque  composant  du  systdme  multi-senseurs  est  reprdsentd 
a  l’dcran  par  une  icone  qui  indique  le  type  du  composant, 
ainsi  que  son  dtat  d’  activitd.  La  symbologie  adoptde  est  la 
suivante : 

Radar  :  Le  secteur  angulaire  observd  en  azimut  est  figurd 
en  blanc. 

Camdra  :  Le  secteur  angulaire  observd  en  azimut  est  figurd 
en  rouge. 

Moyen  de  ddfense :  H  est  reprdsentd  par  un  cand  rouge 
plein 

Position  it  protdger  :  Elle  est  reprdsentde  par  un  cand  blanc 
en  trait  dpais. 
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Logique  de  fusion :  Symbolist  par  des  lignes  bleues 
orient6es  par  une  fleche  reliant  les  capteurs  aux  noeuds  de 
fusion  et  aux  moyens  de  defense. 

Resea  u  physique :  Matdrialisd  par  des  lignes  vertes  dont 
l’dpaisseur  caractdrise  le  type  de  ligne. 

Le  rdseau  virtuel  indique  la  logique  ^association  des 
informations.  Les  noeuds  de  fusion  sont  repidsentds  par  un 
carre  bleu  dans  lequel  le  type  de  fusion  est  symbolisd  par 
un  signe  en  blanc.  Les  signes  sont  L  pour  un  noeud  de 
combinaison  lindaire  des  observations,  +  pour  un  noeud  & 
seuil  haut  (test  ou)  et  Ft  pour  un  noeud  &  seuil  bas  (test  et). 
Le  choix  de  ces  trois  types  de  noeuds  est  justifid  dans  [5]. 
n  n’ arrive  qu’une  seule  ligne  it  un  moyen  de  ddfense  (une 
seule  ddcision  pour  un  point  donnd  de  l’espace :  ddtection 
ou  non  ddtection  d’une  cible  en  ce  point).  Ceci  permet  de 
ddfinir  la  notion  de  sous-systeme  comme  dtant  1’  ensemble 
des  capteurs  qui  conhibuent  a  donner  une  ddcision  a  un 
meme  moyen  de  ddfense.  Par  contre  d’un  capteur  peuvent 
partir  plusieurs  lignes  vers  des  noeuds  diffdrents.  Les  noeuds 
indiquent  le  type  de  fusion  logique  des  ddcisions  (ou  des 
observations)  et  symbolisent  un  terme  ou  un  facteur  dans 
une  dquation  de  fusion. 

Le  rdseau  virtuel  ddcrit  done  de  fagon  graphique  les 
opdrations  de  fusion  et  de  ddcision  de  rdaction  des  ddfenses 
sans  tenir  compte  explicitement  du  rdseau  de 
communications  physique  existant 

Dans  [5],  nous  avons  considdrd  trois  groupes  type  de  lignes 
de  communication : 

-  Les  lignes  haut  ddbit  qui  permettent  la  transmission  de 
quelques  sdquences  viddo.  On  peut  citer  parmi  ces  lignes 
les  fibres  optiques,  (30  Mdga-octets  par  seconde)  ou 
Ethernet  (1 0  Mdga-octets  par  seconde) 

-  Les  lignes  moyen  ddbit,  qui  permettent  la  transmission 
d’une  imagette  au  voisinage  d’une  menace  potentielle  ou 
d’une  carte  radar.  Citons  par  exemple  les  liaisons  RS232 
dont  le  ddbit  est  de  l’ordre  de  50  Kilo-octets  par  seconde. 

-  Les  lignes  bas  ddbit  permettent  de  transmettre  des  valeurs 
numdriques,  position,  variance,  confiance  de  quelques  plots, 
leur  ddbit  est  de  l’ordre  de  500  octets  par  seconde. 

En  pratique,  dans  le  module  SITUATION  du  logiciel 
ROSACE,  nous  avons  intdgrd  la  possibilitd  de  tracer 
interactivement  trois  types  de  liaisons  pour  lesquelles  on 
indique  dans  la  base  de  donndes  le  ddbit  et  la  portde. 

-  Type  1  :  Cables  coaxiaux,  fibres  optiques. 

Ces  liaisons  sont  traedes  au  sol  comme  le  chemin  qu’elles 
empruntent  La  longueur  est  limitde  par  la  portde  indiqude 
dans  la  base  de  donndes. 

-  Type  2 :  Liens  radio  (par  exemple  le  poste  PR4G  de 
Thomson,  autorise  des  transmissions  sdcurisdes  a  4800  bits/s 
pour  data,  phonie  etc...  Les  ddlais  d‘ acquisition  de  cibles 
sont  infdrieurs  a  1  seconde. 


-  Type  3  :  Liens  optiques  pour  lesquels  on  indique  en  plus 
la  hauteur  au  sol  des  transducteurs.  Si  il  y  a  un  masquage 
entre  les  deux  objets,  alors  l’opdrateur  est  averti  de 
l’impossibilitd  de  la  liaison  optique. 

Ces  liaisons  sont  visualisdes  par  des  traits  verts  dpais  pour 
le  type  1 ,  des  tirets  pour  le  type  2  et  des  lignes  en  pointillds 
pour  le  type  3. 

Notons  que  la  ddsactivation  d’une  ligne  du  rdseau  physique 
peut  se  traduire  par  la  suppression  des  dldments  du  rdseau 
logique  qui  ne  sont  plus  compatibles  avec  la  topologie  et  les 
ddbits  du  rdseau  physique  encore  actif.  D’ autre  part  la 
cohdrence  entre  les  seuils  de  ddtection  et  done  les  taux  de 
fausse  alarme  des  capteurs  et  le  rdseau  physique  devra  etre 
systdmatiquement  vdrifide.  II  faudra  aussi  respecter  les 
capacitds  de  transmission  et  de  traitement  en  particulier  dans 
le  cas  des  attaques  "saturantes". 

Pour  configurer  un  systeme  de  veille  ou  de  ddfense, 
l’opdrateur  sdlectionne  done  un  a  un  les  composants,  les 
positionne  et  les  oriente  a  l’aide  de  la  souris,  sur  l’dcran 
reprdsentant  le  thdatre.  Ensuite,  il  spdcifie  les  liens  logiques 
entre  les  diffdrents  composants,  et  ddfinit  le  rdseau  de 
communications. 

Lorsque  l’opdrateur  a  configurd  un  systeme  et  que  cette 
configuration  lui  parait  intdressante,  il  peut  sauver  la 
situation  Sinon  il  peut  la  modifier  par  ajout  d’un  dldment 
de  la  base  de  donndes.  ddplacement,  suppression  ou 
ddsactivation 

La  visibilitd  des  capteurs  figure  2,  la  prdcision  de  localisation 
d’un  hdlicoptere  figure  3,  sa  vulndrabilitd  figure  4  sont 
reprdsentdes  avec  une  dchelle  de  couleurs  sur  un  plan 
d’ altitude  constante,  ici  500m.  Notons  que  par  rapport  a  la 
situation  de  la  figure  1 ,  le  radar  nord-ouest  a  dtd  ddplacd. 


Figure  2 :  Visibilite  a  500m 
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Figure  3  :  Vulnerabilite  d  un  helicoptere 

2.3  Applications  et  perspectives 


Initialement  ddveloppd  pour  la  conception  de  systemes 
de  surveillance  sol-air,  les  applications  du  logiciel  ROSACE 
peuvent  trouver  leurs  sources  dans  d’autres  contextes. 
Signalons  que  le  CEM  (Centre  d’  Essais  de  la  Mdditdrannde) 
a  montrd  un  intdret  pourf  outil  dans  le  cadre  de  la  preparation 
d’ essais.  Une  adaptation  ergonontique  du  logiciel  pour  les 
besoins  du  CEM  est  en  corns. 

D’ autre  part,  F  outil  peut  servir  de  plate-forme  pour  la  mise 
au  point  et  la  comparaison  d’algorithmes. 

3.  ACHEMINEMENT  DES  INFORMATIONS 

Dans  ce  chapitre  nous  ddcrivons  les  moyens  de  commu¬ 
nication  et  d’dchange  d’ informations  entre  les  diffdrents 
composants  du  systdme  multi-senseurs.  Nous  avons  dans 
[1 0]  distingud  deux  niveaux  de  communication : 

-  le  rdseau  physique  est  constitud  de  tignes  physiques  avec 
leur  ddbit  ddfini  comme  le  nombre  de  messages 


transmissibles  par  seconde  ainsi  que  le  ddlai  de  transmission 
que  F  on  supposera  nul  dans  ce  qui  suit. 

-le  rdseau  virtuel  ddcrit  l’dquation  logique  entre  les 
composants  pour  la  fusion  des  informations. 

3.1  Le  rdseau  physique 

Rappelons  que  les  composants  d’un  systeme  multi- 
senseurs  sont  essentiellement  de  trois  types  :  les  capteurs, 
les  moyens  de  ddfense  et  les  processeurs  de  traitement 
Les  capteurs  effectuent  une  observation  du  milieu  et 
transmettent  une  information  cartes  d’dnergie  en  distance- 
azimut  par  exemple  pour  les  radars  2D,  plots,  etc...  vers  un 
processeur  de  traitement,  via  une  ligne  physique  du  rdseau. 
Les  processeurs  de  traitement  effectuent  une  fusion  des 
informations  afin  d’aboutir  d  une  ddcision  qu’ils  communi- 
quent  aux  moyens  de  ddfense.  On  supposera  dans  toute  la 
suite  que  les  capacitds  de  traitement  ne  sont  pas  limitdes. 
Autrement  dit  la  fusion  n’  est  contrainte  que  par  le  ddbit  des 
lignes  de  transmission. 

Les  informations  pourront  etre  achemindes  via  des  multi- 
plexeurs/ddmultiplexeurs  qui  ne  rdduisent  en  aucun  cas  le 
ddbit  ndcessaire.  La  figure  5  montre  un  exemple  de  rdseau 
physique  pour  un  systdme  comprenant  huit  capteurs  et  deux 
processeurs.  Les  fleches  indiquent  la  direction  de  circulation 
des  informations,  les  ronds  verts  sont  des  multiplexeurs/ 
ddmultiplexeurs.  Les  cands  verts  reprdsentent  les  proces¬ 
seurs  de  traitement  Nous  n’ avons  pas  reprdsentd  de  moyen 
de  ddfense. 


Figure  5 :  Exemple  de  reseau  physique 

3.2  La  logique  de  fusion 


Rappelons  que  dans  [5],  [8]  et  [9]  nous  avons  dtudid 
l’apport  de  la  fusion  d’  informations  pour  la  tache  premiere 
du  systdme  multi-senseurs  c’est-a-dire  la  ddtection 
L’apport  a  dtd  quantifiti  par  Famdlioration  de  la  probabilitd 
de  ddtection  Pd  fonction  du  rapport  signal  d  bruit  pour  une 
probabilitd  de  fausse  alarme  Pfa  fixde.  Par  consdquent  dans 
[10],  nous  avons  considdrd  qu’il  pouvait  y  avoir  en  matidre 
de  ddtection,  trois  opdrations  de  fusion  d’un  intdret  notable. 
La  fusion  optimale  (celle  qui  foumit  les  meilleures 
performances  ou  encore  celle  qui  donne  une  probabilitd  de 
ddtection  Pd  maximale  pour  une  valeur  de  Pfa  fixde  consiste 
d  calculer  un  rapport  de  vraisemblances  multi-dimen- 
sionnelles  et  d  le  comparer  d  un  seuiL  Dans  le  cas  ou  les 
capteurs  peuvent  etre  ddlocalisds,  nous  avons  montrd  que 
ce  type  de  fusion  n’est  pas  praticable.  D’ autre  part,  meme 
dans  le  cas  ou  les  capteuis  sont  colocalisds,  la  fusion  lindaire 
optimale  foumit  quasiment  les  mdmes  rdsultats. 
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La  fusion  lindane  cons  is  te  &  effectuer  une  combinaison 
lindaire  dcs  observations  et  &  comparer  cette  valeur  avec  un 
seuii  Ce  type  de  fusion  ndcessite  done  la  transmission  des 
observations  (on  ne  considerera  que  les  dnergies  en  pratique) 
et  done  des  lignes  a  haut  ddbit.  La  fusion  lindaire  optimale 
entre  deux  capteurs  sdpare  le  plan  des  observations  en  deux 
rdgions  et  Z,  par  une  droite  qui  foumit  parmi  toutes  les 
droites  possibles  la  meilleure  valeur  de  Pd,  Pfa  dtant  fixde. 
La  conjonction  pour  laquelle  un  "et"  logique  est  effectud 
entre  les  ddcisions  dmises  par  deux  capteurs,  et  la  disjonction 
pour  laquelle  un  "ou"  logique  est  alors  effectud,  sont  deux 
opdrations  de  fusion  qui  ne  ndeessitent  pas  un  gxos  ddbit 
Ces  deux  tests  peuvent  etre  optimisds.  Nous  avons  montrd 
dans  [5]  qu’il  existait  un  choix  de  seuils  optimal,  ce  choix 
foumit  la  valeur  maximale  de  Pd,  Pfa  dtant  fixde. 

Dans  le  cas  de  deux  capteurs,  la  figure  6  montre  les  rdgions 
de  ddcisions  de  ces  diffdrents  tests. 


Figure  6 :  Regions  de  decision  pour  les  quatre  tests 


La  notion  d’optimalitd  des  diffdrents  tests  montre  que  dans 
le  module  OPTIMISATION  du  logiciel  ROSACE,  il  est  tout 
^  fait  envisageable  d’optimiser  les  seuils  de  ddtection  pour 
amdliorer  les  performances  du  systeme  multi-senseurs. 
Pour  un  sous-systeme  du  systeme  multi-senseurs,  la  fagon 
dont  les  observations  ou  les  ddcisions  doivent  etre  fusionndes 
est  donnde  par  une  dquation  que  nous  appellerons  "dquation 
logique  de  fusion".  Cette  dquation  est  reprdsentde  pour 
l’opdrateur  du  logiciel  ROSACE  par  des  lignes  (comme 
nous  1’ avons  ddcrit  dans  [10])  et  en  machine  par  un  arbre 
dont  la  figure  7  illustre  un  exemple. 


Figure  7 :  Exemple  de  logique  de  fusion  pour  la  detection 


L’ dquation  logique  de  fusion  peut  s’dcrire  de  maniere 
condensde 
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+  la  disjonction.  H  faut  noter  que  le  test  lindaire  est  le  plus 
riche  en  informations  et  s’effectue  tou jours  en  premier  si  il 
y  a  lieu. 

Si  ui,  u2...u8  ddsignent  les  observations  ddlivrdes  par  les 
capteurs  alors  on  calcule  les  combinaisons  lindaires  E»  E. 
E  telles  que 


Lj  =  E  U]  +  Eu2  +  Eu3 

■  Lo  =  X4U4  +  X5U5 
L3  =  /.()U6  +  X7U7 

On  les  compare  aux  seuils  Yi,  Y2.  Y3  et  us  est  compard  a  Yt- 
Enfin,  on  effectue  les  opdrations  logiques 


(E  >  Yi)  et  (Lj  >  Y2) 


lou  (E  >  Ys)  ou  (u8  >  Yt) 


On  prend  une  ddcision  de  prdsence  de  cible  si  l’un  des  trois 
termes  vaut  1 ,  done  soit  si  u8  ddpasse  le  seuii,  soit  si  E 
ddpasse  le  seuii  ou  encore  si  a  la  fois  E  etE  ddpassent  leur 
seuii 


4.  COMPATIBILITY  DU  RESEAU  PHYSIQUE  ET  DE 
LA  LOGIQUE  DE  FUSION 


La  compatibilitd  entre  F dquation  logique  de  fusion  et  le 
rdseau  physique  doit  etre  examinde.  La  question  est  de 
savoir  si  l’opdration  de  fusion  est  rdalisable. 

Dans  F  affirmative  on  peut  se  demander  comment,  a  quel 
prix  et  quel  processeur  aura  le  rdsultat  de  l’opdration. 
Ces  problemes  s’apparentent  aux  problemes  de  transbor- 
dements  dans  un  graphe  et  pour  cela  nous  reprenons  des 
dldments  de  la  thdorie  des  graphes  dont  les  ddtails  figurent 
dans  [2]. 

4.1  Eldments  de  la  thdorie  des  graphes 

Un  graphe  G  est  ddtermind  par  la  donnde  d’  un  ensemble 
X  de  sommets  ou  noeuds,  et  d’un  ensemble  U  de  couples 
ordonnds  de  sommets,  appelds  des  arcs.  On  notera  alors 
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G=[X,U],  On  dira  que  j  est  un  successeur  de  i  si  il  existe  un 
arc  de  la  forme  u=(i,j).  r(i)  ddsigne  1’ ensemble  des 
successeurs  du  sommet  i. 

Etant  donn6  un  sous-ensemble  de  sommets  S,  on  definit 
co+(S)  ensemble  des  arcs  ayant  leur  extrdmitd  initiale  dans 
S  et  leur  extrdmitd  finale  dans  X-S,  et  co"(S)  ensemble  des 
arcs  ayant  leur  extrdmitd  terminale  dans  S  et  leur  extrdmitd 
initiale  dans  X-S. 

La  matrice  d’ incidence  sommets-arcs  associdc  it  un  graphe 
G=[X,U]  est  une  matrice  A=(alu)  avec  i=l...N,  u=l...M  h 
coefficients  entiers  0,  +1,  -1  telle  que  chaque  colonne 
correspond  &  un  arc  de  G  et  chaque  ligne  h  un  sommet  de 
G  avec 

Si  u=(i,j)  alors  aiu=+l  et  aju=-l 

Les  autres  teimes  de  la  colonne  u  dtant  nuls. 

De  maniere  dquivalente,  si  on  considers  une  ligne  i 
quelconque  (conespondant  au  sommet  i)  alors 

co*(i)  =  {u;  aiu=+l} 
co'(i)  =  {u;  aiu=  —1 } 

4.2  Chemin  dans  un  graphe 

Un  chemin  de  longueur  q  est  une  suite  de  q  arcs 
u,,u2,...uq  avec 

u,=(io,ii) 

.  U2=(ii,i2) 

Uq=(iq-1,iq) 

Un  graphe  est  dit  connexe  si  pour  tout  couple  de  sommets 
i  et  j,  il  existe  un  chemin  de  i  a  j.  Dans  le  cas  d’  un  graphe 
non  connexe,  on  associe  a  chaque  sommet  i  sa  composante 
connexe  c’est-zl-dire  F  ensemble  des  sommets  j  tels  qu’il 
existe  une  chaine  de  i  il  j. 

La  verification  de  la  connexitd  ou  la  recherche  des 
composantes  connexes  d’un  graphe  est  l’un  des  premiers 
problemes  de  la  thdorie  des  graphes.  Il  a  une  importance 
vitale  pour  F  elaboration  des  rdseaux  eiectriques, 
teiephoniques. 

Dans  le  contexte  des  systemes  multi-senseurs,  nous 
considererons  les  sommets  particuliers  que  sont  les 
processeurs  et  nous  chercherons  h  connartre  F  ensemble  des 
capteurs  susceptibles  de  transferer  leuis  informations  vers 
un  meme  piocesseur.  Autrement  dit,  nous  chercherons  la 
composante  connexe  d’un  poste  de  traitement  Notons 
toutefois  que  plusieurs  postes  de  traitement  peuvent 
appaitenir  a  une  meme  composante  connexe.  Dans  ce  cas  il 
faudra  determiner  celui  qui  effectuera  le  traitement (probieme 
de  F  allocation  des  ressources). 

La  figure  8  represente  le  graphe  associe  au  systeme 
multi-senseurs  de  la  figure  5.  Les  processeurs  sont  PI  et 
P2,  les  capteurs  SI,  S2,...S8  et  enfin  les  concentrateurs 
n,...I6.  Ces  notations  seront  explicitees  ulterieuremenL 


Graphe  associe  au  systeme  de  la  Figure  5 


Nous  ne  presentons  pas  l’algorithme  classique  qui  permet 
de  trouver  la  composante  connexe  associde  &  un  sommet. 
Cet  algorithme  est  du  a  Tr6maux  1 882  et  Taijan  1 972.  Il 
explore  les  sommets  en  profondeur  d’abord  le  plus  loin 
possible  dans  le  graphe  sans  former  de  cycle,  puis  il  remonte 
jusqu’&  la  demiere  bifurcation  laissde  de  cotd  et  ainsi  de 
suite  jusqu’au  retour  au  sommet  de  ddpart.  L’ ensemble  des 
sommets  ainsi  rencontrds  forme  la  composante  connexe.  Si 
tous  les  sommets  ont  6t6  rencontrds,  le  graphe  est  connexe. 
La  complexity  de  F  algorithme  est  en  O(M). 

H  existe  d’ autre  part  des  algorithmes  qui  permettent  de 
trouver  le  chemin  le  plus  court  entre  deux  sommets.  Dans 
le  cas  simple  ou  les  longueurs  des  arcs  l(u)  valent  toutes  1 , 
il  existe  un  algorithme  en  O(M). 

4.3  Flots  sur  un  graphe 

La  notion  de  flot  sur  un  graphe  foumit  un  modele  gdndrai 
applicable  &  une  grande  varidtd  de  problemes  :  circulations 
du  courant  dlectrique,  r6seau  de  transport,  structuration  des 
rdseaux  de  communications  etc... 

Soit  G=  [X,U]  un  graphe  connexe  dont  les  arcs  sont 
numdrotds  u  =  1 ,2,...M.  Un  flot  dans  G  est  un  vecteur  a  M 
composantes 
$  =  (4*1 » 4*2  >—  <t>M) 

de  I  Rm  tel  que  en  tout  sommet  iG  X  de  G,  la  premiere  loi 
de  Kirchhoff  soit  vdrifide 

^  <t)u=  "X  4>u 

uG  co+(i)  uG  af(i) 

La  composante  (j)u  du  vecteur  (j)  est  appelde  flux  sur  Fare  u. 
La  relation  exprime  la  loi  de  conservation  des  flux  aux 
noeuds  (la  somme  des  flux  entrant  en  un  sommet  est  6gale 
h  la  somme  des  flux  sortant). 

On  appelle  rdseau  de  transport  un  graphe  G=[X,U]  dans 
lequel  chaque  arc  u  de  U  est  muni  d’  un  nombre  cu  ;>  0  appeld 
la  capacity  de  F  arc.  Lorsqu’  on  fait  circuler  un  flot  sur  G,  ce 
nombre  indique  la  limite  supdrieure  du  flux  admissible  sur 
Fare  u.  On  supposera  que  les  capacitds  sont  des  nombres 
entiers  et  que  c„  ^  0. 

Etant  donnys  deux  sommets  particuliers  de  X  dans  le  graphe 
G,  s  la  source  et  p  le  puits,  (s  *  p),  on  considdre  le  graphe 
G°  =  [X,U°]  ddduit  de  G  en  ajoutant  Fare  (p,s)  de  retour  de 
flot  On  convient  de  lui  attribuer  le  numdro  0.  Les  arcs  de 
G°  sont  done  numdrotds  0,1 ,2,...M. 
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On  dit  que  le  vecteur  <|)=(<t>1,...<j>M)T  est  un  flot  desitp  dans  On  a  ainsi  un  graphe  G’  et  on  dit  que  l’on  a  mis  le  probleme 
G  si  les  lois  de  conservation  aux  noeuds  sont  vdrifides  pour  sous  forme  canonique.  Dans  G’  le  probleme  revient  k  trouver 
tous  les  sommets  de  G  sauf  aux  sommets  s  et  p  ou  l’on  a  un  flot  de  valeur  (j>0  entre  s  et  p  tel  que 


X/  <t>u  -  4>»  -  <t>o 

u£  co+(s)  u£  co'(p) 


4>0  est  la  valeur  du  flot  Si  <)>  est  un  flot  de  s  &  p  dans  G  de 
valeur  <]>0  alors  4)’=(<])o4)i>---,))m)T  est  simplement  un  flot  dans 
G°. 

Le  probleme  de  la  recherche  du  flot  maximum  dans  G 
revient  a  determiner  un  Hot  4>’  dans  G°  vdrifiant  les 
contraintes  de  capacitds  pour  u  £  { 1  et  tel  que  <j>0  soit 
maximum.  L‘  algorithme  de  Ford  &  Fulkerson  (1 956)  permet 
de  rechercher  un  flot  maximum 


<t>o -  ✓ ,ai -  s' ,hj 

de  cout  minimum  et  compatible  avec  les  capacitds  a;  sur  les 
arcs  (s,i)  et  bj  sur  les  arcs  (j,p). 

L’ algorithme  de  Ford  &  Fulkerson  (1 962)  permet  de  trouver 
un  flot  optimum  compatible. 

4.5  Discussion 

Les  dldments  de  k  fhdorie  des  graphes  sont  directement 
applicables  pour  l’dtude  de  la  compatibilitd  du  rdseau 
physique  et  de  la  logique  de  fusion. 


4.4  Probldme  de  transbordement 

Soit  G=[X,U]  un  graphe  orientd,  supposd  connexe.  On 
suppose  que  G  a  N  sommets.  On  distingue  parmi  les  sommets 
de  G  des  sommets  sources  et  des  sommets  puits.  Un  sommet 
i  est  une  source  si  il  dmet  une  quantitd  de  donndes  a,  et  un 
sommet  j  est  un  puits  si  il  requiert  une  quantitd  de  donndes 
bj  pour  effectuer  une  opdration.  Pour  le  probldme  qui  nous 
conceme,  les  capteurs  constituent  les  sources  du  graphe  et 
les  processeurs  de  traitement  les  puits. 

Soit  Xs  1’ ensemble  des  sommets  source  et  XP  F  ensemble 
des  sommets  puits.  On  supposera  dans  la  suite  que: 
Xs  n  XP  =  0 

mais  G  peut  possdder  d’autres  types  de  sommets  ni  source 
ni  puits  (c’est  le  cas  par  exemple  les  concentrateurs !) 
Les  quantitds  a,  et  bj  sont  supposdes  dtre  des  entiers  et  de 
plus  dans  un  premier  temps : 

X*, 

i£  Xs  j£  XP 

A  chaque  arc  u=(ij)  du  graphe,  on  associe  un  cout  y* 
reprdsentant  le  prix  de  passage  d’une  unitd  de  flot  sur  Fare. 
Les  capacitds  des  arcs  sont  supposdes  infmies.  En  pratique, 
cela  signifie  que  les  lignes  de  communications  sont  choisies 
de  manidre  d  ne  pas  etre  saturdes. 

Le  probleme  que  F  on  se  pose  est  de  trouver  un  flot  (j)  sur  G 
vdrifiant  les  contraintes  aux  sources  et  aux  puits  et  de  cout 
total  minimum  done  tel  que : 

(j)u  =  U;  V  i  £  Xs 

u£co+(i) 
u£  CO'Ci) 

On  peut  transformer  le  probldme  en  une  recherche  de  flot  d 
cout  minimum  sur  un  graphe  ne  possddant  qu’une  seule 
source  et  qu’un  seul  puits.  Pour  cela  on  ajoute  d  G  un 
sommet  s  "super-source"  relid  d  tous  les  sommets  i  de  Xs 
par  des  arcs  (s,i)  de  capacitd  aj  et  de  cout  nuL  On  ajoute  un 
sommet  p  "super-puits"  relid  a  tous  les  puits  j  de  XP  par  un 
arc  (j,p)  de  capacitd  bj  et  de  coutnuL  On  ajoute  implicitement 
F  arc  de  retour  (p,s). 


Dans  un  premier  temps,  il  faut  trouver  les  composantes 
connexes  du  graphe  associd  au  rdseau  de  communication  du 
systeme  multi-senseurs.  Seuls  les  capteurs  qui  font  partie 
d’  une  meme  composante  connexe  pourront  "fusionnei"  leurs 
informations. 

Un  graphe  peut  etre  connexe,  mais  il  suffit  qu’  une  liaison 
particulidre  soit  coupde  pour  retirer  cette  propridtd.  De  tels 
ardtes  sont  appeldes  des  isthmes.  Il  existe  un  algorithme 
permettant  de  les  trouver. 

Si  Fare  (12,14)  est  saturd  ou  ddtdriord,  le  graphe  se  retrouve 
avec  deux  composantes  connexes  et  deux  groupes  de 
capteurs  se  distinguent  alors :  SI ,  S2,  S3,  S4  en  liaison  avec 
PI  d’une  part,  S5,  S6,  S7,S8en  liaison  avec  P2d’ autre  part 
De  meme,  plus  simplement  les  capteurs  S3  et  S8  peuvent 
se  retrouver  isolds  suite  a  une  coupure  de  la  liaison  (S3,P1 ) 
ou  (S8.P2). 

Par  contre  les  arcs  (11 ,12)  ou  (I5,P2)  ri  ont  aucune  incidence 
sur  la  connexitd  du  graphe. 

Il  apparart  done  ndeessaire  de  favoriser  les  cheminements 
de  donndes  qui  riempruntent  pas  des  isthmes. 

En  second  lieu,  dans  le  cas  ou  les  capacitds  des  lignes  sont 
supposdes  nettement  supdrieures  aux  besoins,  un  algorithme 
permet  de  trouver  (si  celui-ci  existe)  le  flot  solution  de  cout 
minimal  au  probldme  de  transbordement  Le  cout  d’une 
liaison  peut  dtre  une  fonction  de  la  fiabilitd  de  la  transmission, 
ou  du  temps  de  transmission. 

Si  la  composante  connexe  considdrde  contient  plusiems 
processeurs,  on  pourraappliquerl’ algorithme  d  ces  diffdrents 
puits  et  assigner  finalement  le  traitement  au  processeur  qui 
foumit  le  cout  minimal.  Mais  cependant,  il  riest  pas 
ndeessaire  qu’un  seul  processeur  effectue  l’intdgralitd  du 
traitement  Pour  F  exemple  considdrd  dans  le  chapitre 
prdeddent  on  peut  envisager  d’  appliquer  F  algorithme  pour 
chaque  "fusion  dldmentaire".  Elies  seront  impldmentdes 
alors  sur  les  processeurs  qui  foumiront  le  cout  minimal 
Ainsi  une  premiere  allocation  sera  effectude.  On  proedde 
ainsi  en  paicourant  les  diffdrents  niveaux  de  F  arbre  logique. 
Si  le  nombre  de  processeurs  est  faible,  cette  euristique 
convient  sinon  sa  complexitd  croit  vite. 

5.  CONCLUSION 

Ce  document  a  prdsentd  l’outil  ROSACE  dmanat  de 
l’dtude  "Mdthodologie  multi-senseurs  en  ddfense  anti- 
adrienne"  mende  pour  le  compte  du  GC/DA  de  la  DRET. 
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En  second  lieu,  le  probleme  original  de  la  compatibilit6 
entre  logique  de  fusion  et  iriseau  de  communications 
physique,  a  6t6  abondd  Meme  si  il  ri  a  6t6  que  partiellement 
traitd  nous  estimons  qu’il  est  crucial  et  qu’il  ouvre  la  voie 
vers  un  theme  de  recherche  riche. 

En  effet,  il  est  ndcessaire  de  contraindre  les  algorithmes  qui 
irialisent  les  taches  de  detection,  de  pistage  ou  de  classi¬ 
fication,  a  la  structure  du  rdseau  physique  done  aux 
possibility  de  transmission  et  d’dchange  des  informations 
au  sein  du  systeme  multi-senseurs. 

L’  optimisation  des  algorithmes  (afin  d’  obtenir  les  meilleures 
performances)  ne  peut  se  faire  sans  tenir  compte  de  la 
fiabilitd  des  lignes,  des  capacity  et  des  ddlais  de  trans¬ 
mission,  du  cout 
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DEVELOPMENT  OF  A  WIDEBAND  AIRBORNE  LASER  DATA  LINK 


Robert  A.  Gill  and  Robert  J.  Feldmann 
United  States  Air  Force  Research  Laboratory 
WL/AAZP 

Wright-Patterson  AFB  OH  45433-7303 


Summary:  The  United  States  Air  Force  Research 
Laboratory  is  conducting  a  program  to  develop  a 
wideband  airborne  laser  data  link.  The  program 
successfully  ground  demonstrated  a  1.1 
gigabit/second  (sec)  full  duplex  data  Mink  over  a 
distance  of  150  kilometers  (km).  This  ground 
demonstration  was  accomplished  in  Hawaii  in 
September  1995.  The  system  used  in  the  ground 
demonstration  has  been  redesigned  and  will  be 
installed  in  two  jet  aircraft  for  flight  demonstration 
at  distances  up  to  500  km.  These  demonstration 
flights  will  begin  in  September  1998. 

Introduction:  Modern  battlefield  strategy  is 

predicated  on  knowing  where  the  enemy's  (or 
potential  enemy's)  assets  are  located  and  their 
operational  capability.  This  vital  information  is 
constantly  being  gathered  and  updated  by 
various  ground,  space,  and  airborne  sensors.  The 
requirement  to  send  ever  increasing  amounts  of 
tactical  military  information  between  sensor 
aircraft  and  information  processing  facilities  has 
begun  to  press  the  limits  of  present  airborne  data 
links,  even  when  data  compression  techniques  are 
used.  Utilization  of  optical  data  links  is  under 
consideration  by  the  United  States  Air  Force  and 
development  of  a  possible  airborne  laser  data  link 
is  under  way  by  the  United  States  Air  Force 
Research  Laboratory  Avionics  Directorate 
(AFRL/WL/AA).  The  first  part  of  this  development 
process  was  a  successful  ground  demonstration  in 
September  1995  of  a  full  duplex  1.1  gigabit/sec 
laser  data  link  over  a  distance  of  150  km  between 
Mt.  Haleakala  and  Mt.  Mauna  Loa  in  the  state  of 
Hawaii.  This  demonstration  serves  as  the  basis  for  a 
three  year  AFRL  development  program  begun  in 
August  1996,  which  will  culminate  in  a  series  of  air- 
to-air  flight  demonstrations  beginning  in 
September  1998. 

Background:  The  feasibility  of  laser  airborne  data 
links  was  demonstrated  in  the  mid-80's  by  the  AFRL 
HAVE  LACE  (Laser  Airborne  Communications 
Experiment)  Program.  This  program  developed 


and  tested  two  laser  communications  terminals 
that  operated  at  19.2  kilobits/sec.  The  terminals 
were  tested  using  two  KC-135  aircraft  that 
nominally  flew  at  20,000  to  25,000  feet  (ft)  altitudes 
with  separation  distances  out  to  160  km.  The  most 
significant  result  of  the  HAVE  LACE  flights  was  the 
difficulty  of  initial  signal  acquisition  between  the 
two  moving  platforms,  since  it  had  to  be 
performed  manually.  However,  once  signal 
acquisition  was  accomplished,  tracking  proved  to 
be  robust  and  communications  performance  was 
consistently  measured  at  106  bit  error  rate  (BER)  or 
better. 

Since  the  HAVE  LACE  program,  laser  terminal 
development  and  data  rates  have  improved 
dramatically.  The  Ballistic  Missile  Defense 
Organization  (BMDO)  Innovative  Science  and 
Technology  (1ST)  office  has,  since  1988,  invested  in 
the  development  of  laser  communications 
technology  primarily  for  space  based  laser 
crosslinks.  Two  laser  communications  terminals 
(LCT)  built  by  ThermoTrex  Corp.  for  BMDO  under 
that  development  effort  were  the  basis  of  the 
ground  laser  data  link  system  demonstrated  in 
September  1995  in  Hawaii. 


Figure  1 .  Hawaii  Test  Location 

Ground  Laser  Data  Link  System  Description:  The 

laser  data  link  demonstration  equipment  on  Mt. 
Haleakala  consisted  of  a  ThermoTrex  Corp.  built 


Paper  presented  at  the  AGARD  SPP  Symposium  on  “Multi-Sensor  Systems  and  Data  Fusion  for  Telecommunications, 
Remote  Sensing  and  Radar”,  held  in  Lisbon,  Portugal,  29  September  -  2  October  1997,  and  published  in  CP-595. 
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LCT  along  with  its  associated  control  electronics,  a 
Lockheed  Martin  Corp.  built  Common  Data  Link 
(CDL)  Airborne  Modem  Assembly  (AMA),  a  Litton 
Corp.  built  P-IIIA  Inertial  Navigation  Unit  (INU),  a 
McFadden  Systems  Inc.  built  Model  61 2A  hydraulic 
motion  base  simulator,  and  electromagnetic 
shakers  (along  with  their  control  electronics)  which 
were  provided  by  the  AFRL.  The  laser  data  link 
equipment  on  Mt.  Mauna  Loa  was  similar  to  the 
Mt.  Haleakala  equipment  setup  but  it  was 
mounted  on  a  fixed,  stationary  base.  The 
equipment  installation  on  Mt,  Flaleakala  provided 
the  motion  and  vibration  spectrum  to  simulate 
operation  from  a  high  altitude  aircraft,  while  the 
Mauna  Loa  installation  served  to  simulate  a  fixed 
ground  terminal.  The  LCT  was  the  major 
developmental  item  of  interest  for  the  ground 
demonstration  while  the  other  equipment  had 
been  readily  available  for  several  years  and  were 
considered  nondevelopmental  items.  The 
integration  of  this  equipment,  however,  into  a  high 
data  rate  laser  data  link  system  was  unique  to  this 
ground  demonstration. 


telescope  where  it  was  focused  onto  a  200 
micron  pinhole,  which  acted  as  a  field  stop.  The 
light  was  relayed  from  the  pinhole  to  optics 
located  on  the  back  of  the  secondary  mirror  that 
demultiplexed  the  signals  and  imaged  them  onto 
low  noise  EG&G  SLIK  avalanche  photodiodes. 


Figure  3.  Ground  Demo  Laser  Communication 
Terminal 


Figure  2.  Motion  Base  Configuration 

Ground  Laser  Communications  Terminal 
Description:  The  ground  demonstration  LCT 
contained  the  various  optical  apertures  for 
transmitting  and  receiving  the  optical  signals  and 
they  were  mounted  in  a  single  two  axis  gimbaled 
composite  structural  framework. 

The  receive  telescope,  a  14  centimeters  (cm) 
diameter  Schmidt-Gassegrain  design  with 
spherical  primary  and  secondary  mirrors,  collected 
the  incoming  combined  laser  signals  of  810  and 
852  nanometers  (nm)  wavelength.  A  flat 
dichromatic  mirror  at  the  hole  in  the  primary  mirror 
split  the  high  speed  communication  signal  (810 
nm)  and  the  wavelength  locked  beacon  signal 
(852  nm)  and  focused  them  onto  their  respective 
detectors.  Light  at  810  nm  was  reflected  from  the 
dichromatic  mirror  towards  the  front  of  the 


Light  at  852  nm  was  passed  through  the 
dichromatic  mirror  and  directed  to  the  rear  of  the 
telescope  through  an  atomic  line  filter  and  was 
imaged  on  a  charge  coupled  device  (CCD) 
camera.  The  critical  element  in  the  operation  of 
the  LCT  wide  field  of  view  (FOV),  1.25°,  laser 
communication  signal  acquisition  scheme  was  the 
use  of  an  atomic  line  Faraday  filter  which  was 
placed  in  front  of  a  Dalsa  256  x  256  pixel,  225  Flertz 
(Hz)  frame  rate  tracking  CCD  camera.  The 
Faraday  filter  bandwidth  is  0.019  nm,  with  an  in 
band  transmission  of  70%  (using  polarized  light) 
and  an  out  of  band  rejection  of  better  than  10-5. 
Use  of  the  Faraday  filter  required  that  the  two  LCT 
beacon  lasers  be  locked  to  the  transmission  peak 
wavelength  of  the  filter,  852  nm.  This  was 
accomplished  by  temperature  and  current 
stabilization  and  optical  feedback  through  a 
reference  atomic  line  filter.  There  were  two  50 
milliwatt  beacon  diode  lasers  and  four  140 
milliwatt  peak  communications  diode  lasers,  two 
of  which  were  left  hand  circularity  polarized  and 
two  of  which  were  right  hand  circularity  polarized, 
for  transmitting  laser  signals  from  the  terminal.  This 
provided  two  independent  optical  channels  for 
high  speed  data,  each  channel  capable  of  over 
500  megabit/sec  for  a  cumulative  data  link  rate  of 
over  1  gigabit/sec.  The  beacon  lasers  had  a 
divergence  of  2  milliradians  while  the 
communication  lasers  had  a  divergence  of 
approximately  100  microradians.  All  of  the 
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apertures  were  coaligned  and  moved  together 
when  the  gimbals  were  moved. 

The  gimbaled  framework  was  moved  in  azimuth 
and  elevation  by  two  direct  drive  pancake  DC 
motors  through  two  23  cm  angular  contact 
bearings.  A  Canon  angular  shaft  position  optical 
encoder  with  a  1  arcsecond  (4.484  microradians) 
resolution  sampled  at  350  millisecond  intervals  was 
used  to  derive  gimbal  position.  A  Type  li  control 
loop  design  was  used  to  accomplish  signal  source 
tracking.  There  were  two  nested  loops,  an  inner 
stability  loop  (which  also  provided  inertial  pointing 
correction)  and  an  outer  CCD  tracking  loop). 

The  inner  stability  loop  incorporated  a  two  axis 
angular  rate  gyro  which  stabilized  open  loop 
gimbal  movements  and  provided  a  correction 
signal  for  maintaining  a  constant  line  of  sight 
gimbal  orientation  while  tracking  motion  that  had 
a  higher  bandwidth  than  the  CCD  centroid  data. 
The  outer  control  loop  was  compromised  of  the 
tracking  CCD  camera,  its  associated  thresholding 
electronics,  and  a  software  implemented  centroid 
calculator  and  position  algorithm.  This  digital 
signal  processor  error  signal  was  low  pass  filtered, 
scaled,  and  summed  into  the  inner  loop  to  provide 
low  bandwidth  (less  than  3  Hz)  pointing  correction 
for  the  tracking  system. 


Figure  4.  Tracking  System  Block  Diagram 

The  electronics  associated  with  the  LCT  consisted 
of  a  VME  cage  housing  cards  for  beacon  laser 
control,  communications  laser  drive,  and  digitized 
video  conversion.  A  second  VME  cage  had  the 
tracking  and  gimbal  control  cards,  consisting  of  an 
i960  DSP,  a  Dalsa  CCD  camera  drive,  a  CCD 
threshold  and  storage,  a  Cannon  optical  encoder 
interface,  a  motor  amplifier,  an  analog  filter  and 
gyro  interface,  and  a  Radisys  486  imbedded  PC  for 
user  and  communications  interface.  There  were 
two  single  rack  mounted  C-Cor  units  that 


performed  the  digitized  video  multiplexing  to  560 
megabits/sec  and  demultiplexing  before  digital  to 
analog  conversion  for  one  optical  channel. 
Another  rack  mounted  unit  provided  a  repeating 
bit  pattern  at  560  megabits/sec  and  had  a  bit 
pattern  receiver  for  testing  the  second  channel. 
An  interface  to  the  CDL  system's  INU  processor 
provided  azimuth  and  elevation  pointing  angles 
based  on  ephemeris  data  to  the  Mt.  Haleakala 
LCT  for  initial  pointing  prior  to  optical  tracking.  The 
LCT  disregarded  the  INU  pointing  information  when 
the  system  was  optically  tracking  the  signal  from 
the  other  terminal.  The  critical  direction  for  the 
high  bandwidth  data  transmission  for  this 
demonstration  was  from  the  simulated  "aircraft"  to 
the  ground,  which  required  high  accuracy 
tracking  since  the  transmitted  communication 
lasers  had  less  divergence  than  the 
communication  receiver's  field  of  view.  The 
tracking  system  was  required  to  remain  pointed  to 
the  Mauna  Loa  terminal  to  within  40  microradians 
(±20  microradians  peak  from  center). 

Ground  Test  Description:  The  Mt.  Haleakala  LCT 
was  mounted  on  a  six  degree  of  freedom  motion 
base  platform.  The  platform  consisted  of  a  table 
supported  by  six  hydraulic  legs  oriented  at  various 
angles.  The  platform  provided  up  to  20°  of  pitch, 
roll,  and  yaw  motions  and  a  limited  amount 
(approximately  30  cm)  of  lateral  motion.  On  top 
of  this  platform  were  three  electromagnetic  linear 
actuators  oriented  at  right  angles  to  each  other, 
all  attached  to  a  single  phenolic  plate  supported 
by  isolation  mounts.  These  linear  actuators 
provided  translational  vibration  motion  with 
respect  to  the  table  of  the  hydraulic  base.  They 
also  created  angular  vibration  due  to  softness  in 
the  platform  mounts  and  the  slight  off  center  of 
mass  mounting  of  the  actuator  pistons  to  the 
plate.  The  hydraulic  platform  provided  a  vibration 
spectrum  of  DC  to  20  Hz,  while  the 
electromagnetic  actuators  operated  from  20  Hz  to 
200Hz.  Motion  sensors  were  mounted  on  the 
phenolic  plate  and  the  table  to  enable  feedback 
control  and  measurement  of  the  platform 
vibration  spectrum  to  verify  its  matching  with  the 
actual  high  altitude  aircraft  electronics  bay 
vibration  spectrum  specified  for  the  ground 
demonstration. 

The  two  Hawaii  mountains  were  chosen  as  the 
demonstration  location  because  of  their  clear 
atmospheric  conditions,  the  long  ground  path  (150 
km),  and  the  high  elevation  (10,000  and  1 1,200  ft) 
of  the  test  sites.  Due  to  the  curvature  of  the  earth 
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over  this  distance,  the  laser  beams  dipped  to 
approximately  8,000  ft  altitude  mid  way  between 
the  two  mountains.  The  demonstration  setup 
commenced  on  24  August  1995  and  because  of 
low  cloud  cover,  link  closure  did  not  occur  until  3 
September  1995  with  all  subsequent  testing 
completed  by  13  September  1995.  At  that  time  of 
year  in  Hawaii,  a  thick  cloud  layer  extending  from 
approximately  3,000  to  8,000  ft  altitude  is 
omnipresent  between  the  islands  of  Hawaii  and 
Maui.  The  actual  link  tests  took  place  at  various 
times  of  the  day  in  an  attempt  to  avoid  these 
unfavorable  weather  conditions  with  operations 
between  4  am  and  10  am  appearing  to  be 
optimum.  Weather  conditions  during  the  tests 
ranged  from  5°  Celsius  and  heavy  winds  at  4  am  in 
the  morning  to  almost  30°  Celsius,  calm  and  sunny 
at  12  noon.  This  morning  atmospheric  heating 
tended  to  raise  the  cloud  layer  to  above  8,000  ft, 
thus  blocking  the  beam  path  and  ending  the  test 
for  the  day. 

Testing  was  initiated  by  directing  the  Mauna  Loa 
LCT  to  point  to  the  Haleakula  LCT  site  while 
simultaneously  energizing  the  motion  base  and 
LCT  on  Haleakula.  The  Haleakula  LCT  would 
initiate  with  an  arbitrary  pointing  direction  which 
was  corrected  by  the  INU  to  point  the  beacon 
lasers  to  within  the  FOV  of  the  Mauna  Loa 
acquisition  receiver  which  then  responded  with  a 
beacon  laser  signal.  Once  the  beacon  laser  signal 
was  detected  by  the  acquisition  receiver,  of  either 
terminal,  the  terminal  was  steered  to  within  the 
FOV  of  the  communications  receiver  and  tracking 
of  the  beacon  signal  was  maintained  within  this 
FOV. 

Ground  Demonstration  Results:  The  laser 
communication  terminal  pointing,  acquisition, 
tracking,  and  reacquisition  in  a  simulated  high 
altitude  aircraft  vibration  and  turning  maneuver 
environment  were  successfully  demonstrated. 
Duplex  operation  of  the  terminals  was  tested  while 
simulating  a  turn  of  20°  roll  and  15°  yaw.  The 
maximum  roll  rate  was  approximately  8°  per  sec. 
Under  the  full  dynamic  motion  of  the  moveable 
platform,  acquisition  of  the  laser  signal  occurred  in 
less  than  2  sec  and  reacquisitions  were  consistently 
performed  in  less  than  0.25  sec.  The  tracking  errors 
were  estimated  to  be  110  microradians  peak  in 
azimuth  and  60  microradians  peak  in  elevation. 
These  numbers  were  somewhat  higher  than  the  80 
microradians  that  was  measured  during  previous 
ground  acceptance  tests  in  the  integration 
laboratory  primarily  because  of  image  blooming 


on  the  CCD  due  to  varying  intensity  of  the 
received  beacon  signal  after  propagation  over 
the  long  atmospheric  path  between  the  islands. 
Even  though  there  was  software  to  compensate 
for  image  blooming,  the  algorithm  proved  to  be 
too  simple  and  produced  a  perceived  error  of  only 
80  microradians  for  large  intensity  blooms. 

The  simulated  aircraft  LCT  on  Mt.  Haleakala  sent 
two  channels  of  high-speed  data.  One  channel 
was  CDL  video  data  at  274  megabits/sec  and  the 
other  channel  was  multiplexed  uncompressed 
digitized  video  at  560  megabits/sec.  The  Mauna 
Loa  LCT  was  simultaneously  transmitting  a  143 
megabits/sec  digitized  video  signal  on  one 
channel  and  a  low  rate  CDL  audio  link  on  the 
other  channel.  The  lasers  were  modulated  using 
on/off  keying  for  the  digitized  video  channel  and 
intensity  modulation  was  used  for  the  CDL 
channel.  No  forward  error  correction  (FEC)  coding 
was  employed  on  either  channel.  There  was  no 
provision  for  measuring  BER  directly  from  the 
received  communication  signal  which  was 
primarily  video.  Estimating  the  link  quality  from  the 
video  display  was  also  difficult  because  of  the 
effects  of  resynchronization  delays  of  the  video 
demultiplexer,  which  was  designed  for  a  fiber 
optic  channel  and  was  not  optimum  for  the 
frequent  burst  errors  which  were  experienced.  The 
average  BER  was  greater  than  ID6.  However, 
there  were  periods  of  exceptionally  good  signal 
quality.  The  LCT  was  designed  to  have  10  dB  of 
signal  margin  to  combat  scintillation  fading. 
Preliminary  analysis  of  the  scintillation  fade 
statistics  as  measured  over  the  propagation  path 
indicates  that  an  additional  15  dB  of  margin  would 
have  been  necessary  to  achieve  a  BER  of  1CF6, 
given  that  no  FEC  coding  was  used.  A  link  margin 
of  25  dB  is  too  large  a  factor  to  achieve  by  simply 
increasing  the  laser  power  (for  cw  diode  lasers) 
and  therefore  the  implementation  of  an  error 
correction  scheme  is  clearly  indicated  in  order 
that  acceptable  BER  performance  be  obtained  in 
such  a  ground  test  environment. 

Ground  Demonstration  Conclusions:  A  formal 
ground  laser  data  link  demonstration  was 
successfully  conducted  for  sponsoring 
governmental  agencies  and  industry  on  7  and  8 
September  1995.  The  dual  channel,  full  duplex 
capabilities  of  the  LCTs  were  demonstrated  during 
a  full  simulated  aircraft  vibration  spectrum 
coupled  to  a  simulated  20°  roll  (8°/sec  rate)  and 
15°  yaw  maneuver  of  the  motion  platform.  Initial 
signal  acquisition,  reacquisition  (following  both 
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short  and  long  term  outages),  and  tracking 
stabilization  were  also  demonstrated. 

The  communications  performance  exhibited 
occurrences  of  large  signal  degradation  due  to 
atmospheric  scintillation  effects.  While  the  signal 
fades  due  to  the  atmosphere  at  the  Hawaii  test 
sites  were  in  excess  of  25  dB,  it  is  believed  that  they 
could  be  mitigated  through  a  combination  of 
increased  signal  power  and  FEC  coding.  The 
ground  test  results  were  sufficient  proof  of  concept 
of  laser  data  link  acquisition  and  tracking, 
However,  it  was  obvious  that  an  LCT  redesign  was 
required  to  accomplish  long  range 
communications  between  two  high  altitude  aerial 
platforms  under  operational  scenarios.  An  analysis 
of  high  altitude  long  horizontal  path  fade  statistics 
was  also  required  to  determine  the  best 
combination  of  signal  and  processing  gain  that 
would  provide  a  robust  communication  link  with 
acceptable  BER  performance. 


Figure  5.  Airborne  LCT  Test  Vehicle 

Airborne  Demonstration  Requirements:  The  next 
phase  in  the  development  of  the  wideband 
airborne  laser  data  link  is  the  design,  fabrication, 
and  flight  testing  of  an  airborne  LCT  which  will 
demonstrate  1.1  gigabit/sec  communications 
between  two  aerial  platforms  at  a  range  of  500 
km.  The  LCT  is  to  have  at  least  a  hemispherical 
field  of  regard  (FOR)  during  laser  data  link 
operation  from  each  platform.  Two  business  size 
jet  aircraft  are  to  be  the  aerial  platforms  and  are 
to  operate  at  up  to  40,000  ft  altitude  for  the  flight 
demonstration.  An  open  and  competitive  AFRL 
contract  solicitation  resulted  in  the  selection  of 
TrexCommunications  Corp.  of  San  Diego,  CA  to 
design  and  fabricate  the  airborne  LCT  and  Veda 
Inc.  of  Dayton,  OH  to  provide  the  vehicle 
integration  and  flight  operations.  The  two 
contracts  were  initiated  in  August  1996  and  the 
LCT  critical  design  review  completed  in  June  1997. 

Airborne  Laser  Data  Link  System  Description:  The 

requirements  to  have  the  LCT  on  a  business  size  jet 


aircraft,  a  T-39A  or  Sabreliner  40  in  this  case,  and 
the  need  for  a  hemispherical  FOR  drove  the  size, 
weight,  and  installation  constraints  of  the  design 
for  the  airborne  LCT.  The  LCT  is  contained  within  a 
ball  turret  mounted  on  the  underside  of  the 
aircraft  fuselage. 


Figure  6.  Ball  Turret  Assembly 

The  48  cm  diameter  aluminum  ball  turret, 
containing  the  LCT,  is  structurally  mounted  in  the 
lower  escape  hatch  opening  of  the  aircraft,  thus 
requiring  minimal  aircraft  modification.  This 
installation  provides  a  slightly  greater  than 
hemispherical  FOR  since  the  LCT  within  the  turret 
has  360°  azimuth  coverage  and  1 1 0°  elevation 
coverage  (20°  above  horizon  to  straight  down  or 
nadir).  The  turret  follows  the  guidance  commands 
to  the  LCT  and  provides  the  large  angle,  coarse 
steering  function  with  a  7  Hz  bandwidth.  Within 
the  turret  is  a  tracking  gimbal  assembly  for  coarse 
tracking  within  its  12°  range  of  motion  and  50  Hz 
bandwidth.  Mounted  within  the  gimbal  assembly 
is  the  actual  LCT  optical  bench  containing  the 
beacon  and  communication  lasers,  fast  steering 
mirror,  the  beacon  and  communications 
detectors,  and  a  black  and  white  viewing 
camera.  The  fast  steering  mirror  has  a  500 
microradians  range  of  motion  with  a  3  kHz 
bandwidth.  This  four-gimbal  axis  design  (six  axis 
when  considering  the  fast  steering  mirror)  allows 
the  LCT  to  track  through  nadir  smoothly  and 
provides  high  bandwidth  tracking  to  overcome 
aircraft  kinematics  and  vibration.  Atop  the  turret 
azimuth  drive  motor  structure,  mounted  on  a  solid 
aluminum  plate,  is  a  Honeywell  H-764G  Embedded 
GPS  Inertial  (EGI)  unit.  The  EGI,  using  GPS  P-Code, 
provides  the  platform  reference  necessary  to 
stabilize  the  LCT  and  the  aircraft  location 
information  necessary  for  initial  data  link 
acquisition. 
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Figure  7.  System  Block  Diagram 

Airborne  Laser  Communications  Terminal 
Description:  The  airborne  LCT  contains  the  various 
optical  apertures  for  transmitting  and  receiving  the 
optical  signals.  They  are  mounted  on  a  composite 
optical  bench  in  a  single  two  axis  gimbaled 
aluminum  structural  framework  which  is  in  turn 
mounted  within  a  two  axis  gimbaled  aluminum 
ball  turret,  The  airborne  LCT  is  very  similar  to  the 
ground  demonstration  LCT  in  that  the  same 
beacon  and  communication  lasers  (852  and  810 
nm)  are  used.  Instead  of  the  14  cm  diameter 
Schmidt-Cassegrain  telescope,  a  14  cm  diameter 
Maksutov  telescope  is  used.  The  optical  bench  is 
a  new  design  and  the  layouts  of  the  optical  paths 
are  different.  Additional  components  are  used  to 
increase  tracking  resolution  and  received  signal 
strength.  An  analysis  of  a  high  altitude  (40,000  ft) 
long  horizontal  path  (400  km)  indicated  that  the 
airborne  LCT  using  the  four  810  nm 
communications  lasers  produces  a  link  margin  of  9 
dB.  Other  wavelengths  (e.g.  one  micron)  were 
analyzed  because  of  their  higher  laser  output 
power  and  commercial  availability.  However, 
because  of  the  need  to  mount  more  complex 
detectors  and  their  associated  expense,  the 
ground  demonstration  LCT  lasers  and  detectors 
were  deemed  preferable  for  the  airborne  LCT 
design. 

The  detection  scheme  for  beacon  tracking  is 
different  in  the  airborne  LCT.  Whereas  the  ground 
demonstration  LCT  used  two  50  milliwatt  852  nm 
beacons  with  a  divergence  of  2  milliradians  for 
both  coarse  and  fine  tracking;  the  airborne  LCT 
uses  one  110  milliwatt  852  nm  beacon  with  a 
divergence  of  6  milliradians  for  coarse  tracking 
and  one  110  milliwatt  852  nm  beacon  with  a 
divergence  of  340  microradians  for  fine  tracking. 
The  coarse  beacon  signal  is  still  received  through  a 
14  cm  diameter  telescope  and  passed  through  an 


atomic  line  filter  onto  a  288  X  384  pixel  CCD 
detector  running  at  100  Hz.  However,  a  separate 
5  cm  aperture  telescope  is  used  to  receive  the  fine 
track  beacon  signal  and  pass  it  off  a  700  Hz 
bandwidth  piezoelectric  fast  steering  mirror  onto  a 
separate  32  x  32  pixel  CCD  detector  running  at  10 
kHz.  The  four  100  milliwatt  810  nm  communication 
laser  transmit  signals  are  also  aligned  by  the  same 
fast  steering  mirror  so  that  the  communication 
signals  are  sent  back  along  the  same  direction  as 
the  received  fine  track  beacon  signal.  This 
coupling  of  the  incoming  beacon  and 
communication  transmit  signals  through  the  same 
high  bandwidth  fast  steering  mirror  provides  quick 
beam  path  correction  and  no  reliance  on  an  error 
feedback  control  loop  with  its  associated  time 
delay.  The  communication  signal  processing  has 
been  redesigned  to  overcome  the  long  signal 
dropouts  due  to  resynchronization  delays  of  the 
video  demultiplexer  encountered  during  the 
Hawaii  ground  demonstration.  The  C-Cor  unit 
used  for  multiplexing/demultiplexing  the  video 
signal  in  the  ground  demonstration  LCT  has  been 
replaced  in  the  airborne  LCT  by  an  IPiTEK  IMTRAN 
CQ-508  unit.  This  unit  provides  resynchronization 
times  in  sub-milliseconds,  thus  improving  the  overall 
BER. 


Figure  8.  Optical  System 

The  position  resolvers  and  drive  motors  of  the 
ground  demonstration  LCT  have  been  upgraded 
for  quicker  response  and  higher  torque.  Flex  pivots 
are  used  instead  of  bearings  on  the  gimbal 
assembly  to  reduce  static  friction.  The  ball  turret 
contains  high  torque  azimuth  and  elevation 
motors  to  quickly  respond  to  steering  commands 
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under  varying  aerodynamic  loads.  The  airborne 
LCT  control  loops  are  more  complex  than  the 
ground  demonstration  LCT  units  because  of  the 
dynamic  aircraft  environment  and  redesign  of  the 
airborne  LCT.  These  multiple  new  control 
algorithms  include  such  functions  as  initial  signal 
acquisition,  platform  stabilization,  turret  positioning, 
coarse  track  steering,  fine  track  steering,  short 
term  search  and  reacquisition,  and  long  term 
reacquisition.  These  algorithms  are  in  the  process 
of  being  written  and  programmed  into  software. 
The  algorithms  and  their  associated  processing  are 
accomplished  in  the  dedicated  System  Processor 
mounted  in  the  LCT  Electronics  Rack  within  the  test 
aircraft.  The  System  Processor  also  serves  as  the 
interface  to  the  onboard  flight  test  instrumentation 
system.  A  separate  digital  processor  is  also 
mounted  in  this  rack  to  accomplish  LCT 
diagnostics. 

Air-To-Air  Test  Description:  Two  T-39A  (or  Sabreliner 
40)  aircraft  will  be  modified  during  the  summer  of 
1998  and  a  LCT  integrated  into  each  aircraft. 
Flight  testing  is  scheduled  to  begin  with  aircraft 
flightworthiness  tests  in  September  1998  at 
Crestview,  FL.  This  will  be  followed  by  a  series  of 
flights  at  Dayton,  OH  and  other  locations  to 
evaluate  beacon  signal  acquisition  and  tracking, 
communication  signal  acquisition  and  quality,  link 
atmospheric  effects,  aircraft  kinematic  and  airflow 
effects,  and  operational  mission  requirements.  The 
flight  testing  is  an  integral  part  of  the  LCT 
development  process;  therefore,  the  LCT  is 
extensively  instrumented  and  each  test  aircraft 
contains  a  complete  and  comprehensive 
onboard  flight  test  instrumentation  system  to 
monitor  in  real-time  LCT  operation  and  the  data 
link  quality.  Each  test  mission  is  expected  to 
consist  of  approximately  two  hours  of  data 
gathering  at  various  test  conditions. 

Initial  air-to-air  data  iink  operations  will  be 
controlled  by  an  onboard  test  engineer  and 
monitored  by  an  onboard  systems  engineer.  A 
pilot  and  copilot/test  director  will  also  be  part  of 
the  onboard  crew.  Later  flights,  such  as  the 
operational  mission  requirements  test  flights,  will 
delete  the  onboard  engineers  as  the  LCT  is 
operated  in  an  autonomous  mode.  The  saving  in 
aircraft  gross  weight  will  allow  higher  altitude  flight 
and  subsequent  longer  link  ranges. 

The  procedure  for  the  conduct  of  each  test  flight  is 
to  first  accomplish  a  ground  checkout  and 
alignment  calibration  prior  to  takeoff.  Safety 


switches  have  been  incorporated  into  the  laser 
power  circuits  to  prevent  autonomous  laser 
operation  on  the  ground  since  the  eye  safe 
distance  is  approximately  1.5  km.  The  ball  turret 
will  be  in  a  stowed  position  with  the  LCT 
inoperative  during  takeoff  and  landing  to  prevent 
turret  window  damage  and  to  decrease  system 
cooling  requirements.  When  the  two  aircraft  have 
reached  their  test  altitude  and  range,  the  crew  of 
the  "master"  test  aircraft  will  request  a  position  fix 
from  the  "slave"  aircraft  via  an  omnidirectional 
radio  frequency  (RF)  data  link.  A  one  time 
message  exchange  of  each  aircraft's  position  and 
velocity  vector  then  takes  place.  This  information 
is  used  by  each  LCT  to  acquire  the  other  aircraft's 
coarse  track  beacon  signal.  A  transition  is  then 
made  to  the  fine  track  beacon  signal  and  link 
communications  is  initiated.  This  acquisition 
sequence  is  expected  to  be  accomplished  in  less 
than  10  sec.  If  a  test  aircraft  loses  the  track 
beacon  signal,  reacquisition  is  performed  using  a 
propagated  vector  of  the  other  aircraft  based  on 
a  track  file.  If  reacquisition  does  not  occur  within 
one  sec,  a  spiral  search  pattern  is  initiated  by  the 
LCT  for  10  sec.  If  there  is  still  no  reacquisition  after 
10  sec,  a  new  RF  transmitted  data  point  is 
obtained  and  the  initial  acquisition  process  is 
repeated.  The  same  RF  data  link  is  used 
continuously  by  each  aircraft's  flight  test 
instrumentation  system  to  display  the  other 
aircraft's  location  to  the  flight  crew.  This  RF  data  is 
not  fed  continuously  to  the  LCT  Systems  Processor 
for  a  track  file;  the  track  file  is  based  only  upon 
information  derived  from  the  track  beacon  signals. 
Later  operational  mission  requirements  test  flights, 
each  aircraft  will  attempt  to  accomplish  beacon 
signal  acquisition  based  upon  a  priori  knowledge 
of  each  aircraft's  location  at  a  designated  time 
and  will  not  utilize  the  RF  data  link  in  the 
acquisition  process.  The  1,1  gigabit/sec  laser 
communication  signals  will  consist  of  four 
video/audio  channels,  a  serial  digital  data  stream, 
and  a  BER  generator  data  stream. 

Expected  Air-To-Air  Demonstration  Results:  The 

data  collected  during  the  test  flights  should  prove 
the  ability  of  a  wideband  airborne  laser  data  link 
to  communicate  in  the  upper  atmosphere  to  500 
km  with  a  BER  of  106.  The  tests  should  provide 
data  on  atmospheric  attenuation  and  beam 
scintillation.  Data  on  the  effects  of  aircraft  airflow 
upon  beam  steering  should  also  be  gathered.  This 
information  will  be  made  readily  available  to  other 
laser  communication  development  efforts  (e.g.  an 
air-to-space  capability). 
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1.  SUMMARY 

A  new  colour  image  fusion  scheme  is  applied  to  vis¬ 
ible  and  thermal  images  of  military  relevant  scenar¬ 
ios.  An  observer  experiment  is  performed  to  test  if 
the  increased  amount  of  detail  in  the  fused  images 
can  improve  the  accuracy  of  observers  performing  a 
detection  and  localisation  task.  The  results  show  that 
observers  can  localise  a  target  in  a  scene  (1)  with  a 
significantly  higher  accuracy,  and  (2)  with  a  greater 
amount  of  confidence  when  they  perform  with  fused 
images  (either  gray  or  colour  fused),  compared  to  the 
individual  image  modalities  (visible  and  thermal). 

2.  INTRODUCTION 

Scene  analysis  by  a  human  operator  may  benefit  from 
a  combined  or  fused  representation  of  images  of  the 
same  scene  taken  in  different  spectral  bands.  For  in¬ 
stance,  after  a  period  of  extensive  cooling  (e.g.  after 
a  long  period  of  rain  or  early  in  the  morning)  the 
visible  bands  may  represent  the  background  in  great 
detail  (vegetation  or  soil  areas,  texture),  while  the 
infrared  bands  are  less  detailed  due  to  low  thermal 
contrast  in  the  scene.  In  this  situation  a  target  that  is 
camouflaged  for  visual  detection  cannot  be  detected 
in  the  visible  bands,  but  may  be  clearly  represented 
in  the  infrared  bands  when  it  is  warmer  or  cooler 
than  its  environment.  The  fusion  of  visible  and  ther¬ 
mal  imagery  on  a  single  display  may  then  allow  both 
the  detection  and  the  unambiguous  localisation  of 
the  target  (provided  by  the  thermal  image)  with  re¬ 
spect  to  the  context  (provided  by  the  visible  image). 
The  abovementioned  line  of  reasoning  is  frequently 
adopted  to  promote  image  fusion,  and  has  resulted 
in  an  increased  interest  in  image  fusion  methods,  as 
is  reflected  in  a  steadily  growing  number  of  publica¬ 
tions  on  this  topic  [8-13],  A  large  effort  has  been 
spent  on  the  development  of  these  new  image  fusion 
methods.  However,  until  now  there  are  no  validation 


studies  that  investigate  the  applicability  domain  and 
the  practical  use  of  these  techniques.  Ultimately  the 
performance  of  a  fusion  process  must  be  measured  as 
the  degree  to  which  it  enhances  a  viewers  ability  to 
perform  certain  practical  tasks.  Preliminary  results 
on  enhanced  detection  of  targets  embedded  into  real 
scenes  [3]  have  shown  potential  benefits  of  fusion. 
The  present  study  is  performed  (a)  to  investigate  the 
conditions  for  which  the  fusion  of  visible  and  thermal 
images  may  result  in  a  single  composite  image  with 
extended  information  content,  and  (b)  to  test  the  ca¬ 
pability  of  a  recently  developed  colour  image  fusion 
scheme  [2-7]  to  enhance  the  situational  awareness  of 
observers  operating  under  these  specific  conditions. 

3.  METHODS 

3.1.  Image  capture 

3.1.1.  Apparatus 

The  visible-light  camera  was  a  Siemens  K235  Charge 
Coupled  Device  (CCD)  video  camera,  with  a  756  x 
581  CCD  chip,  and  equipped  with  a  remotely  con¬ 
trolled  COSMICAR  C10ZAME-2  (Asahi  Precision 
Co.  Ltd.,  Japan)  zoom  lens  (f=  10.5-105  mm;  1:1,4). 
The  infrared  (IR)  camera  was  an  Amber  Radiance  1 
(Goleta,  CA,  U.S.)  Focal  Plane  Array  (FPA)-camera, 
with  an  array  of  256  x  256  pixels,  operating  in  the  3- 
5  pm  (mid-range)  band,  and  equipped  with  a  100  mm 
f/2.3  Si-Ge-lens.  Each  pixel  corresponds  to  a  square 
1.3  min  of  arc  wide  instantaneous  field  of  view.  The 
entire  array  of  256  x  256  pixels  therefore  corresponds 
to  a  field  of  view  of  about  5.6  degrees  wide. 

The  CCD  and  IR  images  must  be  aligned  before  they 
can  be  fused.  The  signals  of  the  thermal  and  visual 
cameras  are  therefore  spatially  registered  as  closely  as 
possible.  A  2nd  order  affine  warping  transformation 
is  applied  to  map  corresponding  points  in  the  scene 
to  corresponding  pixel  locations  in  the  image  plane. 
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3.1.2.  Conditions 

The  recording  period  is  just  before  sunrise,  and  the 
atmosphere  was  slightly  hazy.  The  visual  contrast  is 
therefore  low.  The  thermal  contrast  is  low  because 
most  of  the  objects  in  the  scene  have  about  the  same 
temperature  after  having  lost  their  excess  heat  by 
radiation  during  the  night. 

3.2.  Image  fusion 

The  computational  image  fusion  methodology  is  de¬ 
veloped  at  the  MIT  Lincoln  Laboratory  [2-7]  and 
derives  from  biological  models  of  colour  vision  and 
fusion  of  visible  light  and  infrared  (IR)  radiation. 

In  the  case  of  colour  vision  in  monkeys  and  man,  reti¬ 
nal  cone  sensitivities  are  broad  and  overlapping,  but 
the  images  are  contrast  enhanced  within  bands  by  spa¬ 
tial  opponent  processing  (via  cone-horizontal-bipolar 
cell  interactions)  creating  both  ON  and  OFF  center- 
surround  response  channels  [14],  These  signals  are 
then  contrast  enhanced  between  bands  via  interac¬ 
tions  among  bipolar,  sustained  amacrine,  and  single¬ 
opponent  colour  ganglion  cells  [15,16],  all  within  the 
retina. 

Fusion  of  visible  and  thermal  IR  imagery  has  been  ob¬ 
served  in  the  optic  tectum  of  rattlesnakes  and  pythons 
[17,18],  These  neurons  display  interactions  in  which 
one  modality  (e.g.  ER)  can  enhance  or  depress  the  re¬ 
sponse  to  the  other  sensing  modality  (e.g.  visible)  in  a 
strongly  nonlinear  fashion.  Such  interactions  resem¬ 
ble  opponent-processing  between  bands  as  observed 
in  primate  retina. 

For  opaque  surfaces  in  thermodynamic  equilibrium, 
spectral  reflectivity  p  and  emissivity  e  are  linearly 
related  at  each  wavelength  A:  p{  A)  —  1  -  e(A).  This 
provides  a  rationale  for  the  use  of  both  on-center  and 
off-center  channels  when  treating  infrared  imagery 
as  characterized  by  thermal  emissivity. 

In  the  colour  image  fusion  methodology  the  indi¬ 
vidual  input  images  are  first  enhanced  by  filtering 
them  with  a  feedforward  center-surround  shunting 
neural  network  [19].This  operation  serves  to  (i)  en¬ 
hance  spatial  contrast  in  the  individual  visible  and 
IR  bands,  (ii)  to  create  both  positive  and  negative 
polarity  IR  contrast  images,  and  (Hi)  to  create  two 
types  of  single-opponent  colour  contrast  images.  The 
resulting  single-opponent  colour  contrast  images  rep¬ 
resent  grayscale  fused  images  that  are  analogous  to 
the  IR-depressed  visual  and  IR -enhanced  visual  cells 
of  the  rattlesnake  [17,18], 


To  obtain  a  natural  colour  representation  of 
these  single-opponent  images  (each  being  an  8-bit 
grayscale  image)  we  have  developed  two  alternative 
methodologies  to  choose  from,  based  on  the  relative 
resolution  of  the  visible  and  IR  images. 

In  the  case  where  the  IR  camera  is  of  significantly 
lower  resolution  (i.e.  half  that  of  the  visible  camera), 
the  enhanced  visible  is  assigned  to  the  green  channel, 
the  difference  signal  of  the  enhanced  visible  and  IR 
images  is  assigned  to  the  blue  channel,  and  the  sum 
of  the  visible  and  IR  images  is  assigned  to  the  red 
channel  of  an  RGB  display  [2-6]: 
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where  the  (. . ,)+  indicates  the  center-surround  oper¬ 
ation.  These  channels  correspond  with  our  natural 
associations  of  warm  (red)  and  cool  (blue). 

In  the  case  where  the  IR  and  visible  cameras  are 
of  comparable  resolution,  the  enhanced  sum  of  the 
enhanced  visible  and  IR  images  is  assigned  to  the 
green  channel,  the  difference  signal  of  the  enhanced 
visible  and  infrared  images  is  assigned  to  the  blue 
channel,  and  the  difference  of  the  enhanced  infrared 
and  visible  images  is  assigned  to  the  red  channel  of 
an  RGB  display  [7]: 
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These  transformations  are  followed  by  hue  and  sat¬ 
uration  remapping.  The  result  is  a  more  natural  ap¬ 
pearing  colour  fused  representation.  Because  the  res¬ 
olution  of  the  cameras  used  for  our  experiments  are 
comparable,  this  is  the  methodology  chosen  to  pro¬ 
cess  the  imagery  used  here.  At  MIT-LL  this  approach 
to  image  fusion  has  been  implemented  on  a  dual-C80 
hardware  platform  capable  of  processing  30  fps  at 
640  x  480  resolution. 

The  grayscale  fused  images  are  produced  by  tak¬ 
ing  the  luminance  component  of  the  corresponding 
colour  fused  images. 


3.3.  Stimuli 

The  stimuli  used  in  this  experiment  are  5  different 
types  of  images: 

-  graylevel  images  representing  the  signal  of  the 
video  (CCD)  camera. 
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-  graylevel  images  representing  the  signal  of  the 
infrared  (IR)  camera, 

-  colour  images  representing  the  result  of  the  fu¬ 
sion  of  corresponding  CCD  and  IR  image  pairs 
(i.e.  the  combination  of  CCD  and  IR  images  of 
the  same  scene  and  registered  at  the  same  in¬ 
stant), 

-  graylevel  images  representing  luminance  com¬ 
ponent  of  the  abovementioned  colour  fused  im¬ 
ages,  and 

-  schematic  graylevel  images,  representing  seg¬ 
mented  versions  of  the  original  visual  (CCD) 
images. 

The  graylevel  images  are  quantized  to  8  bits.  The 
colour  images  are  quantized  to  24  bits  (8  bits  for  each 
of  the  RGB  channels),  and  bitmapped  to  a  256  colour 
map,  adding  some  colour  dither. 

The  individual  images  correspond  to  successive 
frames  of  a  time  sequence.  The  time  sequences  rep¬ 
resent  3  different  scenarios.  These  scenarios  were 
developed  by  the  Royal  Dutch  Army  [20].  They 
simulate  typical  surveillance  tasks  and  were  chosen 
because  of  their  military  relevance. 

The  corresponding  schematic  images  are  constructed 
from  the  visual  images  by 

-  applying  standard  image  processing  tech¬ 
niques  like  histogram  equalization  and  con¬ 
trast  stretching  to  enhance  the  representation 
of  the  reference  contours  in  the  original  visual 
images, 

-  drawing  the  contours  of  the  reference  features 
(judged  by  eye)  on  a  graphical  overlay  on  the 
contrast  enhanced  visual  images,  and 

-  filling  the  contours  with  a  homogeneous 
graylevel  value. 

The  images  thus  created  represent  segmented  ver¬ 
sions  of  the  visual  images. 

Scenario  I  corresponds  to  the  guarding  of  a  UN  camp 
[20],  and  involves  monitoring  a  fence  that  encloses 
a  military  asset.  To  distinguish  innocent  bypassers 
from  individuals  planning  to  perform  subversive  ac¬ 
tions  the  guard  must  be  able  to  determine  the  exact 
position  of  a  person  in  the  scene  at  any  time.  Dur¬ 
ing  the  image  acquisition  period  the  fence  is  clearly 
visible  in  the  CCD  image.  In  the  IR  image  however, 
the  fence  is  merely  represented  by  a  vague  haze.  A 
person  (walking  along  the  fence)  is  clearly  visible  in 
the  IR  image  but  can  hardly  be  distinguished  in  the 
CCD  image.  In  the  fused  images  both  the  fence  and 
the  person  are  clearly  visible.  An  observer’s  situa¬ 
tional  awareness  can  therefore  be  tested  by  asking  the 


subject  to  report  the  position  of  the  person  relative  to 
the  fence. 

Scenario  II  corresponds  to  guarding  a  temporary  base 
[20].  Only  a  small  section  of  the  dune  like  terrain  is 
visible,  the  rest  is  occluded  by  trees.  The  assignment 
of  the  guard  is  to  detect  and  counter  infiltration  at¬ 
tempts  in  a  very  early  stage.  During  the  registration 
period  the  trees  appear  larger  in  the  IR  image  than 
they  really  are  because  they  have  nearly  the  same 
temperature  as  their  local  background.  In  the  CCD 
image  however,  the  contours  of  the  trees  are  correctly 
represented.  A  person  (crossing  the  interval  between 
the  trees)  is  clearly  visible  in  the  IR  image  but  is  rep¬ 
resented  with  low  contrast  in  the  CCD  image.  In  the 
fused  images  both  the  outlines  of  the  trees  and  the 
person  are  clearly  visible.  As  a  result  it  is  difficult 
to  determine  the  position  of  the  person  relative  to  the 
trees  using  either  the  CCD  or  the  IR  images.  The 
fused  images  correctly  represent  both  the  contours  of 
the  trees  and  the  person.  An  observer’s  situational 
awareness  can  therefore  be  tested  by  asking  the  sub¬ 
ject  to  report  the  position  of  the  person  relative  to  the 
midpoint  of  the  interval  delineated  by  the  contours 
of  the  trees  that  are  positioned  on  both  sides  of  the 
person. 

Scenario  III  corresponds  to  the  surveillance  of  a  large 
area  [20].  The  scene  represents  a  dune  landscape, 
covered  with  semi-shrubs  and  sandy  paths.  The  as¬ 
signment  of  the  guard  is  to  detect  any  attempt  to 
infiltrate  a  certain  area.  During  the  registration  pe¬ 
riod  the  sandy  paths  in  the  dune  area  have  nearly  the 
same  temperature  as  their  local  background,  and  are 
therefore  represented  with  very  low  contrast  in  the 
IR  image.  In  the  CCD  image  however,  the  paths  are 
depicted  with  high  contrast.  A  person  (walking  along 
a  trajectory  that  intersects  the  sandy  path)  is  clearly 
visible  in  the  IR  image  but  is  represented  with  less 
contrast  in  the  CCD  image.  In  the  fused  images  both 
the  outlines  of  the  paths  and  the  person  are  clearly  vis¬ 
ible.  It  is  difficult  (or  even  impossible)  to  determine 
the  position  of  the  person  relative  to  the  sandy  path 
he  is  crossing  from  either  the  IR  or  the  CCD  images. 
An  observer’s  situational  awareness  can  therefore  be 
tested  by  asking  the  subject  to  report  the  position  of 
the  person  relative  to  the  sandy  path. 

3.4.  Apparatus 

A  Pentium  100  MHz  computer,  equipped  with  a  Di¬ 
amond  SVGA  board,  is  used  to  present  the  stimuli, 
measure  the  response  times  and  collect  the  observer 
responses.  The  stimuli  are  presented  on  a  17  inch 
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Vision  Master  (Iiyama  Electric  Co.,  Ltd)  colour  mon¬ 
itor,  using  the  640  x  480  pixels  mode  and  a  100  Hz 
refresh  rate. 

3.5.  Procedure 

The  subject’s  task  is  to  assess  from  each  presented 
image  the  position  of  the  person  in  the  scene  relative 
to  the  reference  features. 

In  Scenario  I  the  reference  features  are  the  poles  that 
support  the  fence.  These  poles  are  clearly  visible 
in  the  CCD  images  (Fig.  la),  but  not  represented  in 
the  IR  images  (Fig.  lb)  because  they  have  almost  the 
same  temperature  as  the  surrounding  terrain.  In  the 
(graylevel  and  colour)  fused  images  (Figs,  lc  and  d) 
the  poles  are  again  clearly  visible. 

In  Scenario  II  the  outlines  of  the  trees  serve  to  de¬ 
lineate  the  reference  interval.  The  contours  of  the 
trees  are  correctly  represented  in  the  CCD  images 
(Fig.  2a).  However,  in  the  IR  images  (Fig.  lb)  the 
trees  appear  larger  than  their  physical  size  because 
they  almost  have  the  same  temperature  as  the  sur¬ 
rounding  soil.  As  a  result,  the  scene  is  incorrectly 
segmented  after  quantization  and  it  is  not  possible 
to  perceive  the  correct  borders  of  the  area  between 
the  trees.  In  the  (graylevel  and  colour)  fused  images 
(Figs.  2c  and  d)  the  outlines  of  the  trees  are  again 
correctly  represented  and  clearly  visible. 

In  Scenario  III  the  area  of  the  small  and  winding  sandy 
path  provides  a  reference  contour  for  the  task  at  hand. 
This  path  is  represented  at  high  contrast  in  the  CCD 
images  (Fig.  3a),  but  it  is  not  represented  in  the  IR 
images  (Fig.  3b)  because  it  has  the  same  temperature 
as  the  surrounding  soil.  In  the  (graylevel  and  colour) 
fused  images  (Figs.  3c  and  d)  the  path  is  again  clearly 
visible. 

For  each  scenario  a  total  of  9  frames  is  used  in  the 
experiment.  In  each  frame  the  person  is  at  a  differ¬ 
ent  location  relative  to  the  reference  features.  These 
different  locations  are  equally  distributed  relative  to 
the  reference  features.  Each  stimulus  is  presented  for 
1  s,  centered  on  the  midpoint  of  the  screen,  preceded 
by  a  blank  screen  with  an  awareness  message,  which 
is  presented  for  1  s.  A  schematical  representation 
of  the  reference  features  is  shown  immediately  after 
each  stimulus  presentation  (Figs.  1— 3(f)).  The  posi¬ 
tion  of  the  center  of  the  reference  image  is  randomly 
displaced  around  the  center  of  the  screen  between 
presentations  to  ensure  that  subjects  can  not  use  prior 
presentations  as  a  frame  of  reference  for  detection 
and  localization.  A  complete  run  consists  of  135 


presentations  (5  image  modalities  x  3  scenarios  x 
9  frames  per  scenario),  and  typically  lasts  about  one 
hour. 

The  subject’s  task  is  to  indicate  the  perceived  loca¬ 
tion  of  the  person  in  the  scene  by  placing  a  mouse 
controlled  cursor  at  the  corresponding  location  in  this 
schematical  drawing.  The  subject  has  in  principle  un¬ 
limited  time  to  reach  a  decision.  When  the  left  mouse 
button  is  pressed  the  computer  registers  the  coordi¬ 
nates  corresponding  to  the  indicated  image  location 
(the  mouse  coordinates)  and  computes  the  distance 
in  the  image  plane  between  the  actual  position  of 
the  person  and  the  indicated  location.  The  subject 
presses  the  right  mouse  button  if  the  person  in  the 
displayed  scene  has  not  been  detected.  The  subject 
can  only  perform  the  localization  task  by  memoriz¬ 
ing  the  perceived  position  of  the  person  relative  to 
the  reference  features. 

The  schematic  reference  images  are  also  used  to  de¬ 
termine  the  optimal  (baseline)  localisation  accuracy 
of  the  observers.  For  each  of  the  three  scenarios  a 
total  of  9  baseline  test  images  (Figs.  l-3e)  are  cre¬ 
ated  by  placing  a  binary  (dark)  image  of  a  walking 
person  at  different  locations  in  the  reference  scene. 
The  different  locations  of  the  person  in  these  images 
are  equally  distributed  over  the  entire  reference  inter¬ 
val.  The  image  of  the  walking  person  was  extracted 
from  a  thresholded  and  inverted  thermal  image.  In 
the  resulting  set  of  schematic  images  both  the  refer¬ 
ence  features  and  the  person  are  highly  visible.  Also, 
there  are  no  distracting  features  in  these  images  that 
may  degrade  localisation  performance.  Therefore, 
observer  performance  for  these  schematic  test  im¬ 
ages  should  be  optimal  and  may  serve  as  a  baseline 
to  compare  performance  obtained  with  the  other  im¬ 
age  modalities. 

A  complete  run  consists  of  1 62  presentations  (6  image 
modalities  x  3  scenarios  x  9  frames  per  scenario)  in 
random  order,  and  typically  lasts  about  one  hour. 

3.6.  Subjects 

A  total  of  6  subjects,  aged  between  20  and  30  years, 
serve  in  the  experiments  reported  below.  All  subjects 
have  normal  (or  corrected  to  normal)  vision,  and  no 
known  colour  deficiencies. 

3.7.  Viewing  conditions 

Viewing  is  binocular.  The  experiments  are  performed 
in  a  dimly  lit  room.  The  images  are  projected  onto 
the  screen  of  a  CRT  display.  This  screen  subtends  a 
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viewing  angle  of  25.5  x  19.5  degrees  at  a  viewing 
distance  of  0.60  m. 

4.  RESULTS 

Figure  4  shows  that  subjects  are  uncertain  about  the 
location  of  the  person  in  the  scene  for  about  20%  of 
the  visual  image  presentations  and  22%  of  the  ther¬ 
mal  image  presentations.  The  (graylevel  and  colour) 
fused  images  result  in  a  smaller  fraction  of  about  13% 
“not  sure”  replies.  The  lowest  number  of  “not  sure” 
replies  is  obtained  for  the  baseline  reference  images: 
only  about  4%.  This  indicates  that  the  increased 
amount  of  detail  in  fused  imagery  does  indeed  im¬ 
prove  an  observer’s  subjective  situational  awareness. 

Figure  5  shows  the  mean  weighted  distance  between 
the  actual  position  of  the  person  in  each  scene  and  the 
position  indicated  by  the  subjects  (the  perceived  po¬ 
sition),  for  the  visual  (CCD)  and  thermal  (IR)  images, 
and  for  the  graylevel  and  colour  fusion  schemes.  This 
Figure  also  shows  the  optimal  (baseline)  performance 
obtained  for  the  schematic  test  images  representing 
only  the  segmented  reference  features  and  the  walk¬ 
ing  person.  A  low  value  of  this  mean  weighted  dis¬ 
tance  measure  corresponds  to  a  high  observer  accu¬ 
racy  and  a  correctly  perceived  position  of  the  person 
in  the  displayed  scenes  relative  to  the  main  reference 
features.  High  values  correspond  to  a  large  discrep¬ 
ancy  between  the  perceived  position  and  the  actual 
position  of  the  person.  In  all  scenarios  the  person 
was  at  approximately  300  m  distance  from  the  view¬ 
ing  location.  At  this  distance  one  pixel  corresponds 
to  11.4  cm  in  the  field. 

Figure  5  shows  that  the  localisation  error  obtained 
with  the  fused  images  is  significantly  lower  than  the 
error  obtained  with  the  individual  thermal  and  visual 
image  modalities  (p=0.0021).  The  smallest  errors 
in  the  relative  spatial  localisation  task  are  obtained 
for  the  schematic  images.  This  result  represents  the 
baseline  performance,  since  the  images  are  optimal 
in  the  sense  that  they  do  not  contain  any  distract¬ 
ing  details  and  all  the  features  that  are  essential  to 
perform  the  task  (i.e.  the  outlines  of  the  reference 
features)  are  represented  at  high  visual  contrast.  The 
lowest  overall  accuracy  is  achieved  for  the  thermal 
images.  The  visual  images  appear  to  yield  a  slightly 
higher  accuracy.  However,  this  accuracy  is  mislead¬ 
ing  since  observers  are  not  sure  about  the  person  in 
a  large  percentage  of  the  visual  images,  as  shown 
by  Figure  4.  The  difference  between  the  results  for 
the  graylevel  fused  and  the  colour  fused  images  is 


not  significant  (p=0.134),  suggesting  that  spatial  lo¬ 
calisation  of  targets  (following  detection)  does  not 
exploit  colour  contrast  as  long  as  there  exists  suffi¬ 
cient  brightness  contrast  in  the  gray  fused  imagery. 

5.  DISCUSSION 

This  study  investigates  (a)  for  which  conditions  the 
fusion  of  visual  and  thermal  images  results  in  a  single 
composite  image  with  extended  information  content, 
and  (b)  whether  a  recently  developed  colour  image 
fusion  scheme  [2-5,9]  can  enhance  the  situational 
awareness  of  observers  operating  under  these  specific 
conditions  and  using  visual  and  thermal  images. 

Conditions  in  which  fusion  of  visual  and  thermal  im¬ 
agery  are  most  likely  to  result  in  images  with  in¬ 
creased  information  content  occur  around  sunrise.  At 
this  time  the  contrast  of  both  the  visual  and  the  ther¬ 
mal  images  is  very  low.  One  can  construct  other 
scenarios  involving  night  operations  in  which  both 
modalities  are  lacking  in  contrast.  The  visual  contrast 
is  low  around  sunrise  because  of  the  low  luminance 
of  the  sky.  However,  contours  of  extended  objects 
are  still  visible.  After  some  image  enhancement  (like 
center-surround  shunting,  histogram  equalization  or 
contrast  stretching)  even  an  appreciable  amount  of 
detail  can  be  perceived.  Small  objects  with  low  re¬ 
flectance,  like  a  person  wearing  a  dark  suit  or  cam¬ 
ouflage  clothing,  or  objects  that  are  partly  obscured, 
are  not  represented  in  the  visual  image  under  these 
conditions,  and  can  therefore  not  be  detected.  The 
thermal  contrast  is  low  around  sunrise  because  most 
of  the  objects  in  the  scene  have  about  the  same  tem¬ 
perature  after  losing  their  excess  heat  by  radiation 
during  the  night.  As  a  result  the  contours  of  extended 
objects  are  not  at  all  or  incorrectly  represented  in 
the  thermal  image.  The  fusion  of  images  registered 
around  sunrise  should  therefore  result  in  images  that 
represent  both  the  context  (the  outlines  of  extended 
objects)  and  the  details  with  a  large  thermal  contrast 
(like  people)  in  a  single  composite  image.  To  test  this 
hypothesis  a  large  set  of  image  sequences  is  captured 
around  sunrise  on  different  days.  The  scenes  used  in 
this  study  represent  3  different  scenarios  that  were  de¬ 
veloped  by  the  Royal  Dutch  Army  [20],  The  images 
are  fused  using  the  recently  developed  MIT  colour 
fusion  scheme  [2-5,9].  Graylevel  fused  images  are 
also  produced  by  taking  the  luminance  component 
of  the  colour  fused  images.  Visual  inspection  of  the 
results  shows  that  the  fusion  of  thermal  and  visual 
images  indeed  results  in  composite  images  with  an 
increased  amount  of  information. 
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An  observer  experiment  is  performed  to  test  if  the 
increased  amount  of  detail  in  the  fused  images  can 
yield  an  improved  observer  performance  in  a  task 
that  requires  a  certain  amount  of  situational  aware¬ 
ness.  The  task  that  is  devised  involves  the  detection 
and  localisation  of  a  person  in  the  displayed  scene  rel¬ 
ative  to  some  characteristic  details  that  provide  the 
spatial  context.  The  person  is  optimally  represented 
in  the  thermal  imagery  and  the  reference  features  are 
better  represented  in  the  visual  imagery.  The  hy¬ 
pothesis  is  therefore  that  the  fused  images  provide  a 
better  representation  of  the  overall  spatial  structure 
of  the  depicted  scene.  To  test  this  hypothesis  sub¬ 
jects  perform  a  relative  spatial  localisation  task  with 
a  selection  of  thermal,  visual,  and  (both  graylevel- 
and  colour-)  fused  images  representing  the  above- 
mentioned  military  scenarios.  The  results  show  that 
observers  can  indeed  determine  the  relative  location 
of  a  person  in  a  scene  with  a  significantly  higher  accu¬ 
racy  when  they  perform  with  fused  images,  compared 
to  the  individual  image  modalities. 

This  study  shows  no  significant  difference  between 
the  localisation  performance  with  colour  fused  im¬ 
ages  and  with  their  luminance  components  (the  de¬ 
rived  graylevel  fused  images).  However,  in  some 
conditions  colour  fused  images  are  easier  to  visually 
segment  than  graylevel  fused  images.  As  a  result, 
colour  coding  may  greatly  improve  the  speed  and 
accuracy  of  information  uptake  [21],  and  fewer  fixa¬ 
tions  may  be  required  to  locate  colour  coded  targets 
[22],  Therefore,  dynamic  tasks  like  navigation  and 
orienting,  that  probably  depend  on  a  quick  and  cor¬ 
rect  scene  segmentation,  may  benefit  from  a  colour 
fused  image  representation. 

It  is  likely  that  the  fusion  of  thermal  and  low-light 
level  imagery  may  yield  an  even  better  observer  per¬ 
formance  over  extended  exposure  times  which  often 
lead  to  exhaustion  or  distraction.  Further  research, 
preferably  involving  dynamic  scenarios,  is  needed  to 
test  the  hypothesis  that  colour  image  fusion  schemes 
can  boost  observer  performance  in  these  tasks. 

6.  CONCLUSIONS 

The  fusion  of  thermal  and  visual  images  registered 
around  sunrise  results  in  composite  images  with  an 
increased  amount  of  detail  that  clearly  represent  all 
details  in  their  correct  spatial  context. 

Observers  can  localise  a  target  in  a  scene  (1)  with  a 
significantly  higher  accuracy,  and  (2)  with  a  greater 
amount  of  confidence  when  they  perform  with  fused 


images  (either  gray  or  colour  fused),  compared  to  the 
individual  image  modalities  (visible  and  thermal). 
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Figure  2.  (a)  Original  CCD,  (b)  original  IR,  (c)  graylevel  fused,  (d)  colour  fused,  (e)  baseline  test,  and  (f) 
reference  images  of  Scenario  II. 
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Figure  3.  (a)  Original  CCD,  (b)  original  IR,  (c)  graylevel  fused,  (d)  colour  fused,  (e)  baseline  test,  and  (f) 
reference  images  of  Scenario  III. 
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Figure  4.  The  percentage  of  image  presentations  in  which  observers  are  uncertain  about  the  relative  position 
of  the  person  in  the  scene,  for  each  of  the  5  image  modalities  tested  (IR,  CCD,  graylevel  fused,  colour  fused, 
and  schematical  reference  images). 


Figure  5.  The  mean  weighted  distance  between  the  actual  position  of  the  person  in  each  scene  and  the 
perceived  position  for  each  of  the  5  image  modalities  tested  (IR,  CCD,  graylevel  fused,  colour  fused,  and 
schematical  reference  images).  The  error  bars  indicate  the  size  of  the  standard  error  in  the  perceived  location. 
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PAPER  No:  42 

DISCUSSOR’S  NAME:  ? 

COMMENT/QUESTION: 

How  are  the  images  registered  finely,  and  what  is  the  robustness  of  the  registration  with 
disturbances? 

If  there  is  a  decay  of  one  or  more  pixels  between  images,  does  it  produce  completely  wrong 
colours  on  the  boundary  of  the  objects? 


AUTHOR/PRESENTER’S  REPLY: 

The  CCD  and  IR  images  must  of  course  be  aligned  before  they  can  be  fused.  The  signals  of  the 
thermal  and  visual  cameras  are  therefore  first  spatially  registered  as  closely  as  possible. 

This  is  effected  by  (a)  choosing  a  set  of  fiducial  points  that  are  visible  both  in  the  thermal  and  in 
the  visual  images  and  (b)  ensuring  that  their  positions  coincide  as  much  as  possible  at  each 
location  over  the  common  field  of  view  of  both  cameras. 

Well-defined  fiducial  points  are  created  by  placing  3  large  plastic  jerrycans  on  the  scene.  The 
cans  are  bright  (white)  and  filled  with  hot  water  so  that  they  are  clearly  visible  in  both  image 
modalities.  They  are  evenly  distributed  over  a  horizontal  row  such  that  the  outer  two  are  just 
inside  the  field  of  view.  The  difference  signal  of  the  CCD  and  IR  cameras  is  displayed  on  a  CRT. 
Using  both  this  difference  image  and  the  fiducial  points  both  the  zoom  ratio  (display  size  of  the 
common  field  of  view)  and  the  optical  axis  of  the  CCD  camera  are  adjusted  to  those  of  the  IR 
camera.  This  tuning  is  iteratively  performed  with  the  fiducial  points  at  different  positions  in  the 
common  field  of  view  (for  different  orientations  of  the  optical  axis  of  the  camera  suite).  When  the 
optimal  horizontal  image  registration  has  been  achieved  there  is  still  a  size  difference  of  about  5% 
in  the  vertical  direction.  The  reason  for  this  is  the  fact  that  the  pixels  of  the  CCD  camera 
represent  rectangular  image  segments,  whereas  those  of  the  IR  camera  correspond  to  a  square 
field  of  view.  A  final  correction  for  this  misalignment  is  performed  by  geometrically  transforming 
and  resampling  the  digitized  video  frames.  By  registering  the  scene  for  3  different  vertical 
positions  (top,  middle,  bottom)  of  the  vitual  line  joining  the  3  jerrycans  (situated  in  the  common 
field  of  view  of  both  cameras)  9  matching  points  are  obtained. 

For  the  scenes  used  in  this  experiment  an  additional  number  of  matching  points  is 
opportunistically  available,  because  a  row  of  street  lamps  is  clearly  visible  from  the  viewing 
location.  The  field  of  view  of  the  camera  covers  a  highway  section  containing  6  lamps.  By 
registering  this  highway  scene  for  3  different  vertical  positions  of  the  row  of  lamps  an  additional 
number  of  18  matching  points  is  obtained.  The  scenes  containing  the  matching  sets  of  fiducial 
points  are  registered  at  the  beginning  of  each  recording  session.  Digitized  frames  from  this 
recording  are  used  to  compute  the  coefficients  of  the  affine  warping  transformation  that  maps 
corresponding  points  in  the  scene  to  corresponding  pixel  locations  in  the  image  plane.  The 
digitized  CCD  images  are  then  warped  and  resampled  so  that  their  pixels  are  in  registration  with 
the  corresponding  pixels  of  the  IR  image.  The  alignment  of  both  cameras  is  not  changed  during  a 
session.  Therefore,  a  single  set  of  match  points  can  be  used  to  register  all  scenes  that  are 
recorded  during  a  session. 
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